Why Spatial Computing Matters for Healthcare

A surgeon doing a breast biopsy has an ultrasound on one screen, a CT on another, and the patient in front of her. While she holds a needle steady, her job is to fuse all three in her head — rotating, scaling, and translating the 2D images until they line up with the 3D anatomy she can see and feel.

Surgeons repeat this act of mental alignment thousands of times a year, and most of the time it works. When it doesn't, the consequences range from a longer procedure to genuine harm.

I spent my PhD trying to take some of that work out of the surgeon's head. I explored the idea through ARTEMIS, a remote guidance system; HoloCPR, built for emergency response; and Situated ultrasound, which helped students learn needle placement. They began with different problems, and I only saw their shared lesson while writing my dissertation.

From cognitive load to cognitive offload

The history of medical guidance is a slow handoff of cognitive work from the user to the medium. Written instructions ask the user to interpret the words and imagine each step. Diagrams reduce some of that work, and video reduces it further. AR can place the guidance on the patient itself. Instead of studying an image, measuring from it, and imagining where a needle will land, the clinician sees the path on the patient.

AR does not remove interpretation, but it can reduce it. With the right overlay, meaning and placement become immediately apparent: the clinician sees what to do and where to do it in the same view.

It is tempting to stop there: build the overlay and let the display do the spatial work. The rest of my PhD was spent learning why the handoff is never that clean.

When the hardware fights you

The hardware got in the way of each system differently. In HoloCPR, users sometimes couldn't find the instructions because the HoloLens's narrow field of view left the spatial checklist outside their visual range without giving them any indication of where to look.

ARTEMIS had a different problem. Its registration was good enough for many annotations, but a remote expert might mark a precise point on the patient and have the local surgeon see it a centimeter away. At that scale, a small perceptual error was a clinical error.

The limit was hardest to ignore in the ultrasound work. Medical students liked seeing the ultrasound image on the patient instead of on a monitor across the room, until they tried to guide a needle through it. The display was off by enough to make it useless. In a procedure that tolerates a millimeter or two, “close enough” is not close at all.

Our measurements explained why. The HoloLens 2 tracks both the head and the eyes, but a 1 mm error in the eye-to-display calibration becomes more than 5 mm by the time the overlay lands on the patient. That is enough to rule out needle work.

Three kinds of cues

When I looked back across the three systems, I noticed something I had missed while building them. Each system became usable only after we added a visual aid suited to the problem, and the aids fell into three groups based on the work they did.

Perceptual cues help you read depth and space: shadows, occlusion, motion parallax, stereo. They mimic how the eye reads the physical world, and most modern rendering engines include them by default.

Interaction cues help you find your way around the AR environment. The HoloCPR compass that always pointed toward the spatial checklist was an interaction cue. It turned “where is the information?” into “follow the arrow.”

Contextual cues were less obvious. They don't carry new clinical information; they give the user a visible anchor for comparing the AR overlay with the physical scene. Misalignment becomes something the user can see and account for.

Contextual cues

The ultrasound prototype shows how this works. Students could see the image but couldn't aim through it because the registration was off. We added a virtual needle in the same coordinate system as the situated ultrasound, so both visualizations inherited the same offset from the physical patient.

Because both visualizations shared the same offset, the display remained usable. The virtual needle still lined up with the anatomy in the ultrasound, even though both visualizations were displaced by an inch from the patient. The display was wrong relative to the world but consistent with itself, and students could use that consistency to guide a needle to the target.

An AR-guided breast biopsy prototype revealed the same pattern. We placed a translucent breast model over the physical phantom. The model carried no new clinical information, but it anchored every other AR overlay to a visible object. When registration drifted, the surgeon could see the drift against the model and work around it.

Designing around error

Medical AR has treated registration accuracy as the gate that every useful system must pass: build better trackers, calibrate more precisely, and wait for the next headset. Better hardware will help, but these studies point to another path. With the right visual anchors, an AR interface can remain useful even when its alignment is imperfect.

Designing for that imperfection broadens the question. We still need to ask, “How do we get sub-millimeter registration?” We can also ask, “What cues help the user work with the millimeters we have?” One is a hardware problem on a multi-year horizon; the other is a design problem we can work on now.

The pattern shows up wherever multiple coordinate systems sit on the same patient: image-guided surgery with CT, MRI, ultrasound, and direct vision, each registered with a different error budget. The systems get useful when they agree with each other, even if none is individually perfect.

“Death by GPS”

Perfect registration is often treated as the end state: align virtual content precisely enough, and the interface problem disappears.^* But accuracy alone cannot tell a clinician whether an overlay still makes sense. Contextual cues can compensate for registration error while making the state of the system observable, which gives the clinician a way to verify its guidance.

Navigation offers a useful analogy. “Death by GPS” describes catastrophic incidents in which people followed turn-by-turn guidance into remote, impassable, or unsafe areas.^† It is easy to blame the driver for trusting the device. But the interface also asks the driver to follow its directions while providing little support for judging them against the road.

Medical AR is less forgiving. Even a well-aligned overlay can lead to a bad decision if the clinician cannot see when the patient and the display begin to disagree. Contextual cues anchor guidance to visible structure in the scene. They let the clinician keep asking, “Do these overlays make sense here?” so errors become visible before they cause harm.

Why spatial computing matters

Surgeons already combine medical images and measurements with the patient in front of them. Much of that integration happens in their heads. Spatial computing can move some of that work into the interface by placing guidance where it applies. Hardware accuracy matters, but it does not determine usefulness on its own. Those same contextual cues can make imperfect systems usable while also making their behavior observable. That observability allows clinicians to verify that the guidance still agrees with the patient.