Eye-tracking company Tobii announced it’s partnering with Prophesee to develop its next-generation of eye-tracking for AR/VR headsets and smart glasses, which will be based on Prophesee’s unique event-based vision system.

In short, Prophesee makes specialized cameras and sensors that work more like human eyes. Instead of recording everything like standard camera, their technology only captures changes, such as movement or light changes, making it faster, more compact and importantly more power efficient. Applications have included self-driving cars, industrial machines, smartphones, but also soon XR headsets and smart glasses with the help of Tobii.

The collaboration is said to combine Tobii’s industry leading eye-tracking platform with Prophesee’s event-based sensor technology, which the companies say will enable them to develop an “ultra-fast and power-efficient eye-tracking solution, specifically designed to meet the stringent power and form factor requirements of compact and battery-constrained smart eyewear.”

SEE ALSO
Disney in Talks with Jim Henson Company to Bring 'The Muppets' to VR

“Event-based vision is a perfect match for the growing demand for low-power, always-on sensing in next-generation wearable devices,” said Luca Verre, co-founder and CEO of Prophesee. “This partnership builds on a strong foundation of collaboration between our teams and expands our joint capabilities into the eye-tracking space. With Tobii’s proven leadership in eye tracking, combined with our experience in event-based sensing, we aim to set a new standard for what’s possible in AR/VR and smart eyewear.”

Emma Bauer, Tobii’s Senior Vice President of Integrations, calls smart glasses “one of the most demanding segments for eye tracking,” as it requires ultra-low power, high performance, and seamless integration into a standard glasses form factor.

Ray-Ban Meta Glasses, Image courtesy Meta, EssilorLuxottica

“Through partnering with Prophesee, we add event-based sensors as a complement to our existing camera technologies, giving our customers even more options and flexibility when designing their products,” Bauer says.

As the global leader in eye-tracking, Stockholm, Sweden-based Tobii has been one of the first companies to supply eye-tracking to consumer VR headsets with its slim and effective eye-tracking modules, allowing for thing like foveated rendering, social VR immersion, and automatic interpupillary distance (IPD) regulation—only three of the many reasons why eye-tracking is a game changer for XR.

Nowadays, you’ll find Tobii eye-tracking tech in a host of modern headsets, including PlayStation VR 2, Pixel Crystal, Play for Dream MR, as well as the 2019-era HTC Vive Pro Eye.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Scott Hayden
Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 4,000 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • XRC

    Interesting to see if they update their PCVR solution with event based sensor and software?

    their current PCVR implementation is very effective for my physiology, the visual indicator (vertical position) and dynamic foveated rendering functions are very useful, although I find automatic ipd generally inaccurate, so set ipd manually so it remains a one time adjustment.

  • Christian Schildwaechter

    TL;DR: this is frickin great, you absolutely want this; it will make eye tracking more precise, less power hungry, usable on mobile and will help ETFR to become more efficient, allowing for much higher resolutions; it mimics how the brightness detecting rod cells in the retina work, giving pixels a memory, so they only report changes, dramatically reducing the amount of/effort to process tracking data.

    This is extremely cool tech. Current eye tracking takes a series of images, compares the color values for each pixel and thereby determines the current position of the pupil. So a 400×400@120Hz nIR tracking camera sends ~19MPixels each second, and the processor has to check for each whether they are black or white, with most of them just being white 120 times per second.

    This of course consumes compute resources, which is why tracking cameras are usually low resolution. 120Hz is actually rather high, Quest hand tracking usually ran at 30Hz, and when Meta 2021 introduced 60Hz hand tracking on Quest 2, apps using it were restricted to CPU performance level 3 and GPU level 2 (out of 4) to prevent overheating.

    But for eye tracking even 120Hz is rather low, because the eyes make very fast, seemingly random side movement to help with 3D vision that look like jitter and only last a few milliseconds. These are (probably) one of the reasons why the motion prediction required for ETFR is such a compute expensive process that so far has kept eye tracking mostly away from mobile HMDs with limited compute/power.

    In Prophesee's event based vision the pixels sort of have a memory of their own, so they know they were white the last time too. Instead of sending all pixel values to the SoC for each frame, these "self-aware" pixels only call in when they detect a change, reducing the amount of data to be analyzed by a factor of 10-100, probably even more for eye tracking that only has to deal with black/white values. This for one reduces power consumption, but also allows to significantly increase the "frame rate". As only changes are reported, the amount of events for a currently not moving eye is the same at 100Hz and 10,000Hz, while current tracking cameras would create 100x the data.

    This is actually how the rod cells in our retinas work. While the cone cells detect color at a rather low frequency of ~20Hz, the rod cells only detect brightness (changes) at a lot higher than 100Hz. Which is why watching a movie at 24FPS in the cinema looks fine, but competitive gamers want 240Hz or more monitors. You cannot actually see things in your peripheral vision, you have to turn your eyes/head to recognize them. But you can notice brightness changes, like a tiger approaching from the left, which will then make you turn your head. So something might be sitting right next to you that you won't notice until it moves, with the rod cells reporting a "the brightness has changed" event to the brain.

    The technology wasn't developed for eye tracking, the intro video on Prophesee's site mentions high frequency machine vibrations as indicators of failure, requiring 10K+Hz for detection as a typical use case. But it should work pretty well for eye tracking in VR, both at the high end by making eye movement prediction more precise through higher tracking frequencies, which in turn will make ETFR more effective by reducing the size of the area rendered in high quality, as well as on the low end in smart glasses by drastically reducing the amount of power required for simple tracking.

    No idea when we will see implementations in HMDs, but they already sell a number of sensors, some of them created in cooperation with Sony, and their SDK is at version 5. So they have been doing this for a while, it is not just a research project that may or may not one day become reality.

    • XRC

      I assumed sachades would cause eye tracking errors ( using Tobii 120Hz on Crystal) but it's been very impressive. Can see the slight increase in CPU load and reduced battery time on the hmd when enabled but otherwise very impressive

  • If you remember well, Ultraleap managed also to make hand tracking with event cameras, before it failed. I tried it at AWE last year, there should be an article on my blog about it