Latency is the delay between head motion and the corresponding virtual world update reaching the eyes.
Too much latency results in images drawn in the right place, but at the wrong time, which can create anomalies.
The magnitude of the anomaly varies with head motion; the faster the head moves, the greater the anomaly.
When the head reverses direction, the anomaly is briefly multiplied and becomes far more apparent.
Again, this destroys the “reality” part of VR.
It’s also a recipe for motion sickness.
So latency needs to be super-low.
Somewhere between 1 & 20 ms total for the entire pipeline from the time head motion occurs through:
Transmitting to the display
Getting photons coming out of the display
And getting photons to stop coming out of the display
Since a single 60 Hz frame is 16.6 ms and latency in a typical game is 35 ms or more – often much more – it will be challenging to get to 20 ms or less.