Quick answer

VR in medical imaging means assembling a series of computed tomography (CT) or magnetic resonance (MRI) slices, stored in the DICOM standard, into a 3D volume and displaying it in a headset. A physician or student can explore a patient's anatomy in space, rotate it, and cut through it with clipping planes. It complements the diagnostic workstation rather than replacing it: a 2024 systematic review describes VR for surgical planning as still experimental, though gradually advancing toward clinical use (Queisner and Eisenträger, 2024).

What is DICOM and where does VR data come from?

DICOM (Digital Imaging and Communications in Medicine) is the international standard for medical images and related information, recognized by ISO as ISO 12052 (DICOM Standard). A DICOM file is not just an image. It also carries metadata: who the patient is, which scanner and settings were used, and the spacing between slices.

Studies are stored in a PACS (Picture Archiving and Communication System), the system a facility uses to archive and share images. Modern systems also expose them through DICOMweb, the web API of the DICOM standard that browsers and applications can use (DICOMweb).

For a VR application, the chain starts simply: fetch the series from PACS, read the metadata, and stack the slices into a volume. Without correct metadata (such as slice spacing) the volume will be distorted, even if every single image looks fine.

How do slices become a 3D image?

There are two main methods, and the choice depends on what you want to show.

Volume rendering displays the values from all slices directly. Each density range is assigned a color and opacity (the transfer function), so you can "hide" soft tissue and show bone, or the other way around. It does not require cutting out structures first, but it is computationally expensive.

A surface model is created after segmentation, which means outlining a chosen structure (bone, vessel, tumor) across the slices. The result is turned into a triangle mesh, often saved as STL. Such a model is light, easy to display, and printable in 3D, but it only shows what was segmented. Segmentation is increasingly assisted by computer vision models, and its output still needs review by a specialist.

AspectVolume renderingSurface model (e.g., STL)
InputThe full DICOM slice seriesSegmentation of a chosen structure
What it showsAll tissue, tuned by the transfer functionOnly segmented structures
Compute costHigh, depends on series sizeLow, depends on triangle count
PreparationTuning the transfer functionSegmentation and its review
Typical useExploring the whole studyPresentation, teaching, 3D printing

Why view scans in VR when 3D monitors exist?

Because VR adds depth and scale that a flat screen cannot. On a monitor a 3D volume is still a 2D picture controlled with a mouse. In a headset you see it stereoscopically, can walk around it, lean in, and slice it with your hand.

The most frequently described uses are:

  • Procedure planning. A systematic review of 46 articles (52 studies) found a positive impact of VR on surgical decision-making and anatomy understanding. In some studies, when surgeons reassessed their plan in VR after reviewing standard images, they changed it in 33-60% of cases (Queisner and Eisenträger, 2024).
  • Education. A meta-analysis of 15 randomized controlled trials (816 students) found that VR moderately improved anatomy test scores compared with other teaching methods (Zhao et al., 2020).
  • Team and patient communication. Discussing a case on a 3D model is easier than on a stack of slices.

The US FDA maintains a public list of authorized medical devices that use AR and VR, naming surgical planning, intraoperative guidance, and rehabilitation among the use cases (FDA). The category is real, and specific clinical uses go through formal review.

What are the limitations of VR in medical imaging?

The evidence is promising but methodologically weak. The authors of the 2024 review point to small samples (around 10 participants per study on average), very different study designs, no standard evaluation, and incomplete technical reporting: many papers did not describe how segmentation or rendering was done (Queisner and Eisenträger, 2024). In the education meta-analysis, studies varied widely, and the authors call for assessing cost-effectiveness and adverse reactions (Zhao et al., 2020).

There are practical limits too:

  • Simulator sickness. Frame drops and motion lag cause discomfort, and the 2024 review notes that 120 frames per second or more is desirable to prevent it.
  • Preparation time. Segmentation and visualization setup take work that plain slice viewing does not.
  • Workflow fit. If opening a study in VR means exporting files and copying them by hand, the tool will not be used.

What do regulations say: when is a VR app a medical device?

Intended purpose decides, not technology. In the EU, software intended to provide information used to take diagnostic or therapeutic decisions falls under the Medical Device Regulation, and Rule 11 in Annex VIII generally classifies it as class IIa or higher (Regulation (EU) 2017/745). The MDCG 2019-11 guidance explains how software is qualified and classified. In the US, the FDA's device framework plays the same role.

In project terms, that means making a choice at the very start:

Intended purposeExampleConsequences
Education, presentation, communicationTeaching anatomy on anonymized studiesNo diagnostic claims, a clearly stated intended purpose
Clinical decision supportPlanning a procedure on a specific patient's dataAssess whether it is a medical device, classify under MDR, run a conformity process
DiagnosisReading a studyCertified diagnostic software, not a typical case for VR

The second layer is data. Health data is a special category of personal data under GDPR (GDPR, Article 9), so the application needs access control, logs, and minimal data on the device. Anonymized studies are the best choice for teaching and presentations.

What are the biggest technical challenges?

From our perspective there are three, and all of them have to be solved at once.

Smooth performance with heavy data. A CT series often has a very large number of slices, and VR has to render a separate image for each eye without stutter. You have to balance volume resolution, rendering method, and simplifications so that interaction stays smooth and the image stays faithful to the study.

Safe navigation across series. Users must always know which study and series they are viewing, its orientation (left and right), and its scale. It is easy to get lost in VR, and an orientation mistake is unacceptable in a medical context.

Ergonomics for specialists. Controls must be simple, sessions short, and the way back to a regular view instant. That is why this kind of project needs a prototype that reaches real users very quickly and comes back with their feedback.

What experience do we have in medical imaging and 3D?

We work at the intersection of two areas that meet in this topic. We build software for medical facilities, VR applications including ones for viewing imaging studies, and 3D reconstructions of real places and objects using Gaussian Splatting. Gaussian Splatting is a technique that reconstructs a 3D scene from photos or video, representing it as a set of semi-transparent ellipsoids (3D Gaussians) (Kerbl et al., 2023). It reconstructs what a camera can see. It does not replace CT or MRI.

On the medical data side, we built Pulse EDM, an electronic medical records system with a PACS archive. DICOM files go to PACS and appear in the patient record with a description, the record shows series and metadata, images open in a DICOM viewer, and 3D scans in STL format can be previewed in the record as well. The stack includes DICOMweb and Three.js. On the security side, national ID numbers are encrypted, the GDPR audit log is append-only, and each client gets a separate instance.

That matters for VR, because an immersive app is only useful once it connects to where the studies already live. If patient data cannot leave the facility, plan processing on your own infrastructure. We show what AI on your own infrastructure looks like on our AI infrastructure page.

How do you start a VR medical imaging project?

With the intended purpose and one scenario, not with a headset.

  1. Define the intended purpose. Education, communication, or clinical decision support. This drives the regulatory requirements.
  2. Pick one study type and one structure. For example, CT of one anatomical region. A narrower scope means a faster prototype.
  3. Plan PACS integration. Ideally through DICOMweb, without manual file exports.
  4. Build a prototype and put it in users' hands within weeks. Test smoothness, orientation, and session length on real studies.
  5. Measure usage and feedback before expanding scope. The same rule applies to any technology project, as we describe in why AI pilots never reach production.

Sources