Vive Tracker Used to Create Pressure Sensitive Photoshop Stylus for VR

2802
13

Developer Jimmy Gray recently posted a short video (see above), showing the latest version of a DIY motion-tracked VR stylus prototype. Using an HTC Vive Tracker mounted above a stylus-shaped unit, a VR stylus can be tracked and rendered, usable in Photoshop and capable of pressure sensitivity.

According to Gray’s tweets, the prototype emulates an HID device using ‘Teensy‘, a very compact USB development board. For now, this part of the device appears to connect directly to the PC via a USB cable, but it could potentially be made wireless, like the Vive Tracker itself, in future iterations.

It’s easy to see where the motivation to start this project came from, as Gray is also an artist, with a few examples of his 3D sketch work in Tilt Brush on his YouTube channel. Like many of these motion-tracked accessory projects, the stylus started life using a Vive motion controller to perform the tracking, and this earlier footage shows the flex of a stylus tip being connected to an input on the Vive controller’s trackpad, acting as the pressure sensitivity gauge.

Tracking a stylus in this way is yet another example of the precision of Valve’s Lighthouse technology, something that was highlighted by Valve themselves in a similar way with the marker board in The Lab. However, the system has its limits, and is not going to be as accurate as using a conventional stylus, but there may be applications where a VR stylus is favourable, especially thanks to the real feedback it can provide.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • Me

    This is like building a blender for elephants. It might work, and even involve nice science behind, but what is the damn point of that thing ? I can’t see a single valid use case.

    • NooYawker

      I think the last paragraph explains it. It’s all experimental. Like the guy who got the tracker working on his rift. But he had to use lighthouses. So, it’s just people seeing if they could.

      • jlschmugge

        I think we get that, and kudos to Mr. Gray for getting this to work. I wont deny that it is clever, but I can’t see how this is practical. I really tried to think where this could be used instead of a pressure sensitive tablet or stylus-enabled touchscreen, so I’m curious where Mr. Gray is going with this.

        How much are Wacom tablets now? I’m lucky to have a Surface 4 if I need to draw, but granted its a smaller screen. This is a long-shot to save a buck with an $800 headset.

        And to Road to VR. Really please let people know that you are posting an article about them. I think this is the second time I saw someone caught off guard. You are a popular enough VR publication that I think people will give you the time for a phone interview.

        • James Gray

          I totally get the skepticism. Especially remembering the pixel resolution of current HMDs, tracking reliability, etc. Yeah, this video was more demonstrating the concept (that I could simulate using a pen display with VR), however other individuals have indicated that they would find this useful. I do plan on showing a more polished video at some point demonstrating some different use cases with a more marketed approach. I don’t really know where it stands as a potentially successful product idea or not, and I don’t want to come off as thinking this is some groundbreaking thing, I jsut feel it could be useful for some occasions.

          But to pique your curiosity a bit, and to give you more clarification as to where I think it *could* go:

          Since that video recording, I implemented a haptic motor driver in it for providing haptic feedback when drawing on a virtual canvas in mid-air. The haptic vibration would simulate the feeling of friction generated from drawing on a surface.

          When using it mid-air, there’s no need to be tied to the specific dimensions of a real-world surface area — this would allow someone to draw in a program like photoshop with the canvas being virtually as big or small they want. Whereas actual consumer hardware devices like Cintiqs and what not are static in terms of surface area and only get so big.

          I could also see a “sub-vr” use case where people use it mainly for the Lighthouse tracking, without an HMD — so they could draw on their computer screens (or on some plain surface) directly, (though not with a rough tip of course which would be damaging to the screen, I will plan to reveal how I plan on enabling this possibly in a future update).

          • Dave Myers

            Are you going to be at VRLA?

          • James Gray

            Unfortunately not, but I plan on getting a booth at the Maker Faire coming up in Detroit soon. I will definitely be looking to attend future events in the west coast, just wasn’t ready for VRLA.

    • James Gray II

      Hey man, it simply started from the idea that I’d love to be able to use a pen display to do some concept art and illustration work. Good pen displays (like Wacom Cintiqs) aren’t too cheap and I already had a Vive, so I had the idea to make basically a virtual pen display where I could draw directly on screen, except the screen is virtual and the stylus positioning is captured using Lighthouse. I was tinkering with just using the Vivemotes as the input, but as an artist who wants to be use an expressive tool like a Wacom, I needed to emulate a stylus, which is why I used hardware to accomplish that. It’s totally unrefined and this one little video doesn’t market the concept well at all. I didn’t even know this article was going to be published! But I did get some positive feedback from people that they’d want to use something like this, so I plan on at least making a better video with a producer friend of mine, to demonstrate what a more polished experience would look like.

    • James Gray

      Hey man, it simply started from the idea that I’d love to be able to use a pen display to do some concept art and illustration work. Good pen displays (like Wacom Cintiqs) aren’t too cheap and I already had a Vive, so I had the idea to make basically a virtual pen display where I could draw directly on screen, except the screen is virtual and the stylus positioning is captured using Lighthouse. I was tinkering with just using the Vivemotes as the input, but as an artist who wants to use an expressive tool like a Wacom, I needed to emulate a stylus, which is why I used hardware to accomplish that. It’s totally unrefined and this one little video doesn’t market the concept well at all. I didn’t even know this article was going to be published! But I did get some positive feedback from people that they’d want to use something like this, so I plan on at least making a better video with a producer friend of mine, to demonstrate what a more polished experience would look like.

      • Paul

        Dude, pay not attention to this guy. Bravo for doing development work. Congratulations. The medium is new and evolving and I think ideas like this are fantastic. The technology and cost will shrink and this is a great step!

      • Jeff

        I think this is a great idea! With one or two extra tips it seems like this could be an awesome way to draw our physical spaces and objects for use in VR. It’s cool as a pen too, but have you guys tried that at all? I would love to draw my play space, add whatever I want, and hang out in it…

      • Hogo

        please keep up the work on this! it’s a really cool idea, and as the resolution of the vive and other VR headsets increases, it will become more and more interesting. It’s really easy for someone to jump on a comment section and spout useless negative crap like “Me” did. A lot harder to try new things and create new experiences. bravo.

    • CazCore

      good thing the world isn’t limited by YOUR imagination limits. :)

    • Michigan Jay Sunde

      That surface he’s drawing on is bigger than a $2,300 Cintiq monitor, while the little contraption he’s using must cost less than $200 to build. And the surface itself is not important – it could be smaller OR much larger. He could be drawing on a wall. He could be drawing on a very large table. He could be drawing on a concrete floor. Imagine military commanders drawing battle plans at a roundtable, or football coaches drawing plays on a miniature field. Or game-masters designing dungeons in miniature. Or VR developers meeting around a huge whiteboard. Or musicians tracking a complex piece of music. Or anyone at all using this as a physical interface for web browsing, shopping, reviewing and retouching photos… drawing the attention of a fantasy pet inside a virtual terrarium, or virtual fish inside a fishbowl. Imagine two of these used in harmony, like Tom Cruise’s gloves in Minority Report. Imagine using this to calibrate the exact boundaries of the furniture inside your tracked space. Imagine using it as a 3D-object digitizer for creating vector-topology from real objects (these machines have been available for CAD – at ungodly prices – for a long time.) This is an awkward and imprecise first version of something that will ABSOLUTELY exist and be commonplace within the next couple of years. I had the idea to do this already but haven’t made any real progress toward building it (and I don’t have access to the tracking puck) but it did inspire me to pickup a 3D printer and start learning Fusion 360. Be careful what you write of as have no “valid use case”. You don’t want to be the first guy who said that about the mouse. ;)