EMG Input

While controller and hand-tracking are proven to be useful input devices for XR today, once we move toward all-day worn headsets that we take into public we’re probably not going to want be swing our arms around or poke the air in front of us just to type a message or launch and application while on the bus.

To that end, Meta has been researching a much more subtle and natural means of XR input via wrist-worn EMG. The company detailed some of its work on a wrist-worn input device earlier this year, and at Connect last week for the first time it showed a prototype device being used for text input.

Though the floating window interface on the right is just a simulation, Meta purports that the left view shows an actual prototype device driving the text inputs.

And while it’s slow today in its prototype form, Abrash believes it has huge potential and room for improvement; he calls it “genuinely unprecedented technology.”

Contextual AI

Meta is also working on contextual AI systems to make augmented reality more useful.

In order to help a system understand the intent of the user, Meta used its detailed virtual apartment recreation as an underlying map to help the glasses understand the user’s intent by observing their gaze.

Quest 3 Pops up in Best Buy Listing With Clues About Resolution, Release Date

By anticipating intent, Abrash says, the user may only need to have a single button for input which chooses the correct action depending upon the context—like turning on a TV or a light. It’s clear to see how this ‘simple input with context’ dovetails nicely with the company’s concept of a wrist-mounted input device.

– – — – –

Abrash says this overview of some of Meta’s latest R&D projects is just a fraction of what’s being invested in, researched, and built. In order to reach the company’s ultimate vision of the metaverse, he says its “going to take about a dozen major technological breakthroughs […] and we’re working on all of them.”


This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • Adam

    Just imagine someone stealing your girlfriends photorealistic avatar… Enjoy all of your coworkers having a swing with her in the “MetaBrothel Extreme Penetration — Cuckold Edition”.

    • ViRGiN

      This spect has been covered by Mark.

    • Plou

      It can already happen. It’s called imagination.

      • silvaring

        email me your imagination archive when you get a chance, I want to download it into my brain.

    • kontis

      You are not describing anything new. It’s not even close to being as big of a deal as you think it is.

      We had photoshop dramas like this in the 90s with just pictures, some wanted to delegalize photoshop and after a few years no one cared anymore.

      Then people started ripping 3D models of real actors from video games 20 years ago and doing nsfw machinima with them. With current high fidelity models it’s even more widespread. Sometimes there is a bit of drama but nothing substantial (and sometimes it’s on official level, like the Ellen Page’s model in Beyond two souls being naked in the developer mode).

      Then we had Deep fakes drama. As before no one cares anymore.

      That “photorealistic avatar” will be nothing but 3D model and never fully accurate and never behaving like real person, so after a few clickbaity articles no one will care, because it will feel like a more elaborate cringy photoshop and just like with the uncanny valley, when people feel something is off they lose interest.

      • Andrew Jakobs

        Uhm actually realistic moving digital characters are already possible at this point, and the last pieces of removing unnatural motions is already being worked on and perfected. In only 2 years or so you really wouldn’t be able to tell the difference, if people want to.

    • Andrew Jakobs

      Well, people downvote you, but it certainly is something that has to be taken into account. Deepfakes already are a problem, but with ultrarealistic avatars it can even go another step further.

    • sfmike

      The real question is will FB/Meta ever allow a MetaBrothel to ever exist or will any expression of adult sexuality get you kicked off the platform like we have seen enforced by FB over the years.

      • ViRGiN

        Valve doesn’t allow any erotica, unless it’s all furry/weeb or nazi fetish. Why would Meta be any different?

    • Anonymous

      I think this is simply destined to happen no matter what as technology improves. It will just be a way of life and not necessarily a bad thing, or maybe even a good thing.

      I will just admit, I look forward to the day it is simple to make deepfakes or fool around in adult ways with the avatar of whatever person I have a fancy for. I’m pretty sure you would try it too if given the choice and it were effortless. As would 99% of other men and some women, though few would admit it.

      Human nature will make it happen.

      • guest

        Covet thou neighbors wife is no longer a comandment when you can also buy a simulation of the inside of their house from Facebook AKA Meta…

  • VRFriend

    Game engines have this tech since 10 years and it did not cost billions of dollars.
    CryEngine, Unreal Engine. C++ code publicly available :) Meta added some more stuff. All these companies copy each other.
    The wall to capture the avatar is such an obvious thing. Costs maybe 30 000 USD to fill with remotely controlled cameras. Nothing new. I worked on this in the year 2000, before Facebook even existed. It is interesting Meta is pushing this, but I do not see anything new here beside connecting this with VR social platform.

    • ViRGiN

      So what are you saying exactly?

      Did you tell them they overpaid about 33333%?

    • kontis

      The things game engines already do was NOT the point of this article or this research and it is NOT where the money went.

      This is like telling an artists that he shouldn’t be proud of his art because paint was invented centuries ago.

    • Andrew Jakobs

      Uhm, bullshit, as an Unreal Engine lover I can honestly say they didn’t have these realistic human avatar tech until they announced Meta humans, which they only added this year, and it certainly wasn’t realtime..

      • VRFriend


    • benz145

      Before shititng on people’s work, perhaps you should know more about it?

      Their innovation in codec avatars is not about creating a photogrammetry model. It’s animating the model well enough to cross the uncanny valley, in-real time, through inputs like face, head, and hand tracking, which is exceptionally difficult. You can learn more about the way the system works here: https://www.roadtovr.com/facebook-expands-on-hyper-realistic-virtual-avatar-research/

  • 144Hz

    This is the beginning of the end.

  • Lucidfeuer

    The only interesting (but very limited) thing I see is the contextual AI, which other just call basic computer vision interfacing. The rest is making them look like Magic Leap, they’re not developing anything substantial, I mean what’s that thing about cloth simulation?

    • Lester

      You may not see it but I certainly do. VR has always been a systems integration and software problem. Otherwise it’s just It’s screens strapped to your eyes. They are solving those problems. I think it will be a huge deal in the long term. But I can’t see it’ll be good for any of us. It is the beginning of a boring, dystopian distraction that our children will be drawn into on a mass scale, not like 1984, but like brave new world. Fake dreams, fake ads, fake jobs, fake life to hide the lowering standards in this one.

  • Arturs Gerskovics

    I personally think they went overboard. Too much hype about things we maybe will have in 10-15 years. They could have spent a little segment on that and the rest about what we can have now or at least VERY soon. At the end of the day, Quest is a mobile VR/AR and it just can’t do the same things what PC can do. So, any cloth simulations or vast high fidelity open worlds will require a decent power which just cant be mounted on your head. It needs an outside source to feed. Maybe META MACHINE could help this. Zuck could make $200-300 META Machines matching RTX3090 and i9 with crazy cooling to drive his fantasy METAVERSE.

    • kontis

      Connect keynotes are no longer for VR nerds and geeks. They are now for investors and general public. They are trying to move to the mainstream with stupid ads. We are far too niche for Facebook.

      It’s a known tactic and it often works. Every time Apple shows new technically absurd (and scientifically incorrect) graph comparing their product to some undefiled competing solution the tech people laugh and cringe how stupid and misleading it is, but general public is in awe and gets hyped like crazy. Blame people for being stupid and eating these things up. Facebook is just falling at execution of the same BS method.

      • psuedonymous@mailinator.com

        The OC/FC events are for VR geeks and nerds who are interested in VR research, not consumer hardware. The consumer hardware fans will get dedicated launch hyper events when those devices are unveiled just like any other electronic device, whilst the VR nerds get Connect showing an overview of research including and beyond the papers already published by FRL.

  • david vincent

    Impressive but once again : who wants to have his IRL face & body as his avatar ?

    • ViRGiN

      Everyone but furries and weebs. Good luck working with real people using your favorite Genshin Impact avatar.

      Who wants to be a furry, hanging out with weebs?

      • John Doe

        You forgot incel gamers xd

        Also, remote working will never be a thing, because companies hate it ;p

    • I’d have to agree, most of us aren’t models. But maybe switching bodies and still retaining realism is possible?

    • Kenny Thompson

      People in professional settings.

  • Mrfox Babbit

    Neckbeards and incels rejoice! The true question lies in can they render a realistic fedora on a jaunty angle and subservient girlfriend AI?

    • John Doe

      Huh, I though video gamers are neckbeards and incels xd

  • Kenny Thompson

    Wow, I didn’t realize that the white room was actually in VR as well until the 3rd watch. Crazy.