The Facebook Reality Labs Research team shared some of its latest audio initiatives today. The group aims to build technologies into an AR headset that will supercharge your hearing by making it easy to isolate the sound of your conversation in a noisy environment, and to be able to reproduce virtual sounds that seem like they’re coming from the real world around you. A custom HRTF (Head-related Transfer Function)—a digital version of the unique way each person hears sound based on the shape of their head and ears—is key to delivering such experiences, but the process is time consuming and expensive. The team is investigating a scalable solution which would generate an accurate HRTF from a simple photograph of your ear.

Facebook Reality Labs (FRL) is the newly adopted name of team at Facebook which is building immersive technologies (including Oculus headsets). Facebook Reality Labs Research (FRLR) is the research & development arm of that team.

Today Facebook Reality Labs Research shared an update on a number of ongoing immersive audio research initiatives, saying that the work is “directly connected to Facebook’s work to deliver AR glasses,” though some of the work is also broadly applicable to VR as well.

Spatial Audio

One of the team’s goals is to recreate virtual sounds that are “perceptually indistinguishable” from the sound of a real object or person in the same room with you.

“Imagine if you were on a phone call and you forgot that you were separated by distance,” says Research Lead Philip Robinson. “That’s the promise of the technology we’re developing.”

In order to achieve that goal, the researchers say there’s two key challenges: 1) understanding the unique auditory characteristics of the listener’s environment, and 2) understanding the unique way that the listener hears sounds based on their physiology.

Understanding the acoustic properties of the room (how sounds echo throughout) can be done by estimating how the room should sound based on the geometry that’s already mapped from the headset’s tracking sensors. Combined with AI capable of estimating the acoustic properties of specific surfaces in the room, a rough idea of how a real sound would propagate through the space can be used to make virtual sounds seem as if they’re really coming from inside the same room.

SEE ALSO
Facebook Says It Has Developed the 'Thinnest VR display to date' With Holographic Folded Optics

Facebook researchers also say that this information could be added to LiveMaps—an augmented reality copy of the real world that Facebook is building—and recalled by other devices in the same space in a way that the acoustic estimation could be improved over time through crowd-sourced data.

The second major challenge is understanding the unique way everyone hears the world based on the shape of their head and ears. The shape of your head and ears doesn’t just ‘color’ the way you hear, it’s also critical to your sense of identifying where sounds are coming from around you; if you borrowed someone else’s ears for a day, you’d have a harder time pinpointing where exactly sounds were coming from.

The science of how sound interacts with differently shaped ears is well understood enough that it can be represented with a compact numeric function—called a Head-related Transfer Function (HRTF). But accurately measuring an individual’s HRTF requires specialized tools and a lengthy calibration procedure—akin to having a doctor test your eyes for a vision prescription—which makes it impractical to scale to many users.

To that end, Facebook Reality Labs Research says it hopes to “develop an algorithm that can approximate a workable personalized HRTF from something as simple as a photograph of [your] ears.”

To demonstrate the work the team has done on the spatial audio front, it created a sort of mini-game where participants wearing a tracked pair of headphones stand in a room with several real speakers scattered throughout. The team then plays a sound and asks the participant to choose whether the sound was produced virtually and played through the headphones, or if it was played through the real speaker in the room. The team says that results from many participants show that the virtual sounds are nearly indistinguishable from the real sounds.

Context-aware Noise Cancellation

While “perceptually indistinguishable” virtual sounds could make it sound like your friend is right next to you—even when they’re communicating through a headset on the other side of the country—Facebook Reality Labs Research also wants to use audio to enhance real, face-to-face conversations.

SEE ALSO
Researchers Develop Method to Boost Contrast in VR Headsets by Lying to Your Eyes

One way they’re doing that is to create contextually aware noise cancellation. While noise cancellation technology today aims to reduce all outside sound, contextually aware noise cancellation tries to isolate the outside sounds that you want to hear while reducing the rest.

To do this, Facebook researchers built prototype earbuds and prototype glasses with several microphones, head tracking, and eye-tracking. The glasses monitor the sounds around the user as well as where they’re looking. An algorithm aims to use the information to figure out the subject the user wants to listen to—be it the person across the table from them, or a TV in the corner of the room. That information is fed to the audio processing portion of the algorithm that tries to sift through the incoming sounds in order to highlight the specific sounds from the subject while reducing the sounds of everything else.

– – — – –

Facebook is clear that it is working on this technology with the goal of eventually bringing it to AR and VR headsets. And while researchers say they’ve proven out many of these concepts, it isn’t yet clear how long it will be until it can be brought out of the lab and into everyday headsets.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • kontis

    Great videos!

    One thing I don’t understand in Facebook’s and Sony’s aspirations to make custom HRTFs for every user is that we already have something working great WITHOUT any customizations: binaural audio or ASMR. There are examples (like the famous virtual baber shop) that blow out of the water the best synthetic HRTF audio, even when using $5 earbuds.

    On the one hand, this is not surprising, because this is like live action video being more photorealistic than CG animation rendered in real-time.

    On the other hand, according to audio specialists this shouldn’t really work so well because we all have different ears and heads.

    Something doesn’t add up here.

    My hypothesis is that binaural audio working so well might be an illusion caused by the realism of every other aspect, but not actual positioning, meaning if you had ot pin point the actual position of audio source it would be affected by the difference between your ear’s shape and the binaural setup. But the doubt still stands – why bother with complicated customization when there are more important areas of audio still not achieving real life quality that are more noticeable to users. The binaural audio’s illusion works great and illusion is what this is all about…

    • Ad

      Michael Abrash, chief scientist at Facebook Reality Labs Research, responded to questions in a briefing call with journalists related to how the use of this technology might affect social norms. Could a personal conversation at a restaurant table be heard by a nearby AR-enhanced patron? Would the nearby patrons ever know their conversation was heard by someone else? And would that conversation be less likely to happen in the first place if people knew mediated hearing was more common than it is at the time of this writing? Abrash and Ravish Mehra, the audio team lead at FRL Research, gave a few examples of some potential mitigation strategies that might be employed to limit the range of this feature in the future. Mehra explained in a prepared statement they intend “to put guardrails around our innovation to do it responsibly, so we’re already thinking about potential safeguards we can put in place…for example, before I can enhance someone’s voice, there could be a protocol in place that my glasses can follow to ask someone else’s glasses for permission.”

    • Miqa

      If you strive for accurate sound reproduction with headphones, binaural recordings are not enough and also feels like it would be a weird way to make sound effects intended for XR. Not enough as in what you will hear and what I will hear will be different due to the shape of our ears.
      I can recommend this article if one wants a deeper dive in different methods used today with their respective advantages and drawbacks. If I summarise it, none of the methods are perfect. A custom HRTF for your own ears and headphones has probably the best potential to fix those issues, IMO.

      https://www.headphonesty.com/2020/04/harman-target-curves-part-1/

  • Ad

    The uploadVR version of this article actually talked about the real point a lot more. Being able to listen to other people’s conversations, and the ability to just remove sounds from the world selectively. Someone homeless on the street asking for help? Gone. A person speaking another language? Gone. A protest outside? Gone. A woman talking about her plans for the day? Listen in like a creep.

    I honestly cannot believe real human beings walk in to work at Facebook, make stuff like this, and are able to sleep at night.

    • silvaring

      Spatial Audio is a truly beautiful technology dude… please don’t be a hater.

      • Ad

        “Michael Abrash, chief scientist at Facebook Reality Labs Research, responded to questions in a briefing call with journalists related to how the use of this technology might affect social norms. Could a personal conversation at a restaurant table be heard by a nearby AR-enhanced patron? Would the nearby patrons ever know their conversation was heard by someone else? And would that conversation be less likely to happen in the first place if people knew mediated hearing was more common than it is at the time of this writing? Abrash and Ravish Mehra, the audio team lead at FRL Research, gave a few examples of some potential mitigation strategies that might be employed to limit the range of this feature in the future. Mehra explained in a prepared statement they intend “to put guardrails around our innovation to do it responsibly, so we’re already thinking about potential safeguards we can put in place…for example, before I can enhance someone’s voice, there could be a protocol in place that my glasses can follow to ask someone else’s glasses for permission.””

  • My Thoughts

    You know what would be great Facebook! If you actually brought out a solid VR headset that is not just the bare minimum to experience VR. The Rift S does not cut it. Any dark scene is so grey it looks like mush.

  • Amazing research! This is Facebook’s way of showing that even if Valve has the best speakers out there now, they have some aces up of their sleeves

  • fuyou2

    Firstly Facebook Should Get-Rid of FB-account requirement for oculus!!! Stop spying on fucking people.. FUYOU2 FACEBOOK……

    • EliteForceCinema

      Are you trying to say you want Oculus to go out of business for good and get all of their employees to go bankrupt and homeless because you think Oculus is owned by Facebook and that you think Facebook is bad? Cause it sounds like you are!

  • TechPassion

    First, you need to log-in to your real name Facebook account. Thanks, but no!