Tracking and latency are just prerequisites.
Once you have good enough tracking and latency, you can draw in the right place at the right time; then you learn about all the other interactions of displays with the human perceptual system.
The key here is that the way displays present photons to the eyes is nothing like the real world, and it’s a miracle we can see anything coherent in displayed images at all.
In the real world, objects emit or reflect photons continuously.
On a display, pixels emit fixed streams of photons for discrete periods of time.
Pixels are also fixed in space relative to the head.
This has major implications; let’s look at a couple of them.
For the following discussion, it will be useful to understand some very simple space-time diagrams, like the one shown here.
The horizontal axis is x position relative to the eyes, and the vertical axis is time, advancing down the slide.
You can think of these diagrams as showing how an object or an image would move horizontally across your field of view as time passes.
You can also find a discussion of space-time diagrams on my blog; I’ll give the URL at the end of the talk.
In this diagram we have an object that is not moving relative to the eyes. The plot is a vertical line because there’s no change in x position over time.
It’s important to understand that the x position in these diagrams is relative to the position and orientation of the eyes, rather than the real world, because the eyes’ frame of reference is what matters in terms of perception. So this diagram could be a case where both the eyes and the object are unmoving, or it could be a case where the eyes are smoothly tracking an object as it moves. In either case, the object would remain in the same x position relative to the eyes.
This diagram shows a real-world object that’s moving from left to right at a constant speed relative to the viewer, while the eyes remain fixated straight ahead – that is, the eyes aren’t tracking the moving object. Here’s an example of the sort of movement this diagram depicts.