The really hard parts.
There are three really hard problems that if solved would make VR far more convincing:
- Tracking
- Latency
- And stimulating the human perceptual system to produce results indistinguishable from the real world
All three are different aspects of the core problem, which is the interaction between the display and the human perceptual system
You may well wonder whether head mounted displays are really so different from monitors.
The answer is yes – and more so than you’d imagine, for two key reasons.
The first reason is that, as I mentioned earlier, in order for virtual images to seem real, they have to appear to remain in the correct position relative to the real world at all times. That means, for example, that as the head turns in this slide, the virtual view has to change correspondingly to show the correct image for the new head position, just as would happen with a real-world view.
This is obviously required in AR, where virtual images coexist with the real world, but it’s required in VR as well, even though you can’t see the real world, because you have a good sense of your orientation, position, and movements even without the help of vision, thanks to hardware and software in your head that’s effectively a gyroscope, accelerometer, and sensor fusion filter.
The second reason is that, unlike monitors, VR displays move rapidly relative to both the real world and the eyes.
In particular, they move with your head.
Your head can move very fast – ten times as fast as your eyes can pursue moving objects in the real world when your head isn’t moving.
Your eyes can accurately counter-rotate just as fast.
That means that if you fixate on something while you turn your head, your eyes remain fixed with respect to the real world, but move very quickly relative to the display – and they can see clearly the whole time.
It’s important to understand this, because it produces a set of artifacts that are unique to head mounted displays.