Steve Jobs once said to a gathering of senior advisers that when it came to the first iPhone, “we’re going to patent it all,” (New York Times reports). While Apple has certainly changed in the years since Job’s passing, the company has undoubtedly continued to aggressively pursue patents, sometimes racking up hundreds of filings in a single month with inventions spanning everything from touch-sensitive smartwatch bands to 3D environmental mapping. This week however, the company was granted two patents that establish basic hardware and software solutions not only pointing towards a prospective Apple AR device, but marking out some fundamental territory in the process.

Both patents, spotted by Apple Insider, were first filed by Metaio GmbH in 2014, a German AR firm later acquired by Apple just a year later.

Patent number 9,560,273, entitled Wearable information system having at least one camera, details a wearable device with a “display attached to his head in front of his eyes” that can use either an onboard SoC or in a wirelessly connected smartphone. The patent goes on to detail the invention’s reactive power mode, letting it switch between a battery-saving low-power mode and an active high-power mode that automatically reacts when a real-world object is matched with a reference object.

apple ar patent
Apple AR patent detailing methods for computing 3D coordinates, image courtesy USPTO

In the figure below, the patent shows off a Google Glass-style configuration that is “especially well suited to be used with head mounted displays and a camera pointed at the space in front of the user.”

Apple’s ‘Google Glass-style’ configuration, image courtesy USPTO

Patent 9,558,581, entitled Method for representing virtual information in a real environment, delves into the software side of AR, and is centered around virtual info overlays and proper occlusion perception.

Audi Co-founded VR Entertainment Startup Holoride Files for Bankruptcy

The setup proposed in the patent paves way for a device that creates a 3D geometry model of the real environment and shows virtual information in both an occlusion mode and a non-occlusion mode, meaning objects are correctly presented to the user even if they’re hiding behind real-world obstacles. And to that, patent ‘581’ focuses on different ways of making virtual information distinct from the real world so users can quickly differentiate the real environment from the superimposed images. The patent proposes a few ways of doing this, including showing real-world objects as semi-transparent when they’re occluded by virtual info (great for driving) and using dashed lines to indicate a separation from the real and virtual.

apple ar software patent
Apple’s method of creating “proper occlusion perception,” image courtesy USPTO

While this all seems fairly basic, these sorts of patents help lay an important defensive framework that can help against future disputes from other manufacturers working in the sector; eg. Apple v. Samsung, Apple v. Microsoft, Apple v. Nokia, the list goes on. And if the recent ZeniMax v. Facebook lawsuit has taught us anything, you don’t ever want to be the one writing that half-billion dollar check after you already shelled out $2 billion for the intellectual property in the first place.

So while not giant revelations on their own, this, alongside Apple’s ever-growing library of AR patents, certainly lends credence to the idea that Apple is investing serious time and money in the field, and it doesn’t look to be ‘just in case’.

Apple’s AR Investments

Just one year after the launch of the iPhone, Apple started patenting head mounted displays with features not unlike the VR headsets of today. Of course, at the time it wasn’t feasible to produce such a unit, but again, patents aren’t really for that purpose anyway.

Apple is Adding Support for Vision Pro's Input System to WebXR

apple virtual reality hmd head mounted display vr headset patentThe company has since patented various AR/VR headset designs, like it’s Gear VR-esque phone-based headset, or the most recent patent in summer 2016 outlining a high FOV AR display.

Apple’s recent acquisitions have also pointed towards a heightened interest in AR. Besides acquiring Metaio, Apple has also taken on marker-less facial mapping and animation firm Faceshift, facial expression recognition & analysis company Emotient, and Flyby Media, a large-scale SLAM, indoor navigation, sensor fusion, image recognition, and 3D tracking company.

Another nail in the coffin: around 2014, Apple started hiring specifically for AR/VR software experts at their Cupertino HQ, first for an ‘App Engineer’ who was tasked to “create high performance apps that integrate with Virtual Reality systems for prototyping and user testing,” and then a month later four additional hires with immersive tech talent, including a ‘VR/AR Programmer’.

As we know it, Apple is a perennial ‘johnny-come-lately’ to established tech, preferring to see markets mature before they enter in with a big bet like the iPhone or Apple Watch, and the company seems famously stand-offish when it comes to VR. According to a report by Vanity Fair, Apple CEO Tim Cook says that virtual reality isn’t actually the way to go in his opinion. “Virtual reality sort of encloses and immerses the person into an experience that can be really cool but probably has a lower commercial interest over time,” said Cook.

So if VR isn’t in Apple’s commercial interest currently, the evidence thus far suggests that the company is heading eye balls-first into augmented reality. And we can’t wait to see what the world renowned electronics giant has in store.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • Get Schwifty!

    I don’t believe we are going to see anything from Apple for maybe two to three years… love to be wrong… but that’s just a gut feeling.

    • No, you’re, unfortuneatly, not wrong.

  • Sebastien Mathieu

    those patent are so vague they are just amaze me… a 10 year old could draw them, no wonder all those companies are going to court for patent violations…….

    • NooYawker

      The amazing part is the patents get granted. There’s something seriously wrong with the process.

  • Sam Illingworth

    “preferring to see markets mature before they enter in with a big bet like the iPhone”

    What market was mature when the iPhone entered? Did a market for that sort of device even exist?

    • Get Schwifty!

      Some people want to claim the Blackberry and Palm led the way…. which in a sense they did, but they were basically nothing but business devices… the iPhone leveraged several concepts together to create a consumer device, just like they did with the iPad and to an extent the Apple Watch. They are usually not first in (like the really old days) but come in at a key point with a product to bring it to the consumer market.

      At one point in the mid 80’s they were a pretty decent gaming platform, but nowadays they don’t seem interested in that outside of making revenue off shit like Candy Crush, etc., so the way the view VR is not from a gamer perspective, but in the AR world which is going to take more time to mature so I doubt anything will appear from Apple for at least a few more years. I personally think without Jobs they will remain a shadow of themselves… Tim Cook, while hitting all the points to be a CEO to me fails in the one thing Jobs had which was real vision… he comes across to me as a caretaker for the legacy of Apple, but little more. That kind of company needs a central visionary with a strong personality because that’s how it was founded…. this is not Cook in the least.

      • Sam Illingworth

        Yes, I agree, the pre-iPhones were a different category, same for iPad. Other people were already doing a similar thing with smart watches though, though that market is mostly the Apple Watch right now, in terms of sales.

        Aye, with their lack of interest in games AR makes more sense for them than VR, though of course VR isn’t just about games.

    • NooYawker

      For some reason many people like to say that about Apple.

      • OgreTactics

        maybe because you must completely obliviously stupid to think otherwise of the Steve Jobsless Apple?

        • NooYawker

          Someone is angry about how total strangers spend their money.

          • OgreTactics

            You mean I’m not a spineless coward hypocrite who’ll stupid fucking people underserving of money get away with being responsible for the shittification, speculation, anti-competitive and anti-innovative strategies of corporations? Yes.

          • JMB

            :-) relax dude

            And people ‘deserving of money’ are those who earn it.

          • OgreTactics

            I’m relax, I’m only relax because I allow myself the freedom to tell a person (whom I have nothing against) has really fucking stupid attitudes, opinions and ideas that he is free to debate or argue with.

  • OgreTactics

    Apple is too late to the party. Their patent make no(new)-sense whatsoever. Google now combines years of the expertise of a smartphone and OS optimised VR headset, with a maturing app/apis/software and a maturing Tango hardware.

    • you are underestimating the power of the … logo. Have no doubt, i can already picture it, people will line up outside Apple stores when their AR product becomes available. Most of those people will have no clue how it compares to the competition. All they will care about is that there is a fruit logo somewhere.

      • OgreTactics

        You underestimate the power of the minds behind and in front of the logo, as well as the power of time.

        You’re relatively right that today exists a mass market consumer base that would buy any shit thrown at them with a satisfactory image, and understandably Apple’s industrial design and UX is still way better than the competition.

        Except for a growing number of them even that doesn’t justify cost, un-innovation, restriction, instabilities, obsolescence…while competitors are starting to have enticing arguments in their design and functionality whether it’s the Galaxy S7/Note + GearVR or the Xiaomi Mix.

  • Sponge Bob

    So Apple is a classical patent troll – patents, patents and no products
    Surprise, surprise

  • Nashoba Darkwolf

    So how long before apple sues Oculus, Valve, and MS for patent infringement?