In an interesting case of software pushing the limits of hardware, Valve recently updated SteamVR Tracking code to account for the speed of high-level Beat Saber players.
At first glance it might not be apparent, but Beat Saber is an excellent challenge for a VR controller tracking system. You might think that a fast-paced VR shooter or sword fighting game would be the game that would be pushing SteamVR Tracking to its limits, but Beat Saber is actually far more demanding.
In a recent update to SteamVR Beta (which will soon roll out to the SteamVR main branch), the following sentence appeared in the patchnotes:
Increase limits of what we thought was humanly possible for controller motion based on tracking data from Beat Saber experts.
Though it seemed like it might have been just a joke, Valve developer Ben Jackson offered up some detail in the comments:
The tracking system has internal sanity checks to identify when things go wrong. For example, if our math says you are behind your only basestation, clearly we made a mistake, because we wouldn’t be getting any signal from behind the basestation. One of these checks relates to how fast we thought it was physically possible for someone to turn their wrist. It turns out that a properly motivated human using a light enough controller could go faster (3600 degrees/sec!) than we thought.
Essentially, Valve built a number of assumptions into the SteamVR Tracking code so that it can identify clearly erroneous data (like a controller signaling that it is moving way faster than should be possible). However, it seems that one of those assumptions was actually too low for the speed of some high-level Beat Saber players, and Valve has now adjusted the code accordingly.
The main reason why Beat Saber manages to push the limits of VR controller tracking so effectively is not just because of the top speed of a player’s hands, but because of the drastic changes in acceleration when players are whipping their hands back and forth to cut distant notes on time.
Most VR controller tracking systems use two tracking methods and combine them together. The first system is an IMU inside each controller which senses rotation and translation, and can do so with low low latency and high frequency. The issue however is that IMU’s are prone to ‘drift’, so they can’t be relied upon alone for tracking. That’s where the other tracking method comes in: in the case of SteamVR Tracking, that’s the external ‘basestations’ which are used to establish the absolute position and rotation of the tracked object and to correct for drift from the IMU. However, the basestations update slowly compared to how fast the IMUs are (in the ballpark of 100Hz vs. 1,000Hz), and until that next positional correction comes in, all the tracking is up to the IMU.
In many cases, the moments of tracking between position corrections is aided by prediction, which uses prior motion to project where the tracked object is heading next. When moving slowly or drawing predictable arcs through the air, this works well. But Beat Saber is something of a worse-case scenario for that prediction because there’s no way for the tracking system to predict when the player is about to suddenly and completely reverse the direction of their controller (which happens constantly in the game, especially at higher levels).
Despite the challenge, SteamVR Tracking has held up extremely well to Beat Saber’s tracking needs except, it seems, in some edge cases which have now been patched. It’s frankly pretty amazing that the system manages to work as well as it does for the game; Valve ought to be pretty happy that they chose to aim for such a high tracking performance threshold from the outset.