HoloLens Hands-On: Building For Windows Holographic

Back in January, HoloLens was a Frankenstein's monster of a headset. In fact, the eyepiece itself was only part of the test rig: the rest was conspicuously tethered to it with a physical cable hooked up to a Windows 10 PC. To say there's been a dramatic shift in hardware in the ninety days or so since then is an understatement. Microsoft brought a fleet of HoloLens prototypes to BUILD 2015, each of them a standalone computer, eyepiece, sensor rig, and spatial sound system all integrated into a single headset, and offered me the opportunity to see what creating a Windows Holographic experience was like.

What hasn't changed are Microsoft's near-obsessional demands around secrecy. Before I was allowed anywhere near HoloLens, I had to sequester my electronics away in a locker the company provided: phone, laptop, even my smartwatch all needed to be stowed away, lest I somehow record my experience on anything more advanced than a pad of paper.

Microsoft remains tight-lipped on what, exactly, is inside HoloLens at the moment, either. The exact identity of the CPU and GPU is a mystery, and the Microsoft rep I asked about battery life told me "I honestly don't know" how long the standalone headset might last.

So what do we know about hardware? Well, from a hardware design standpoint, it's fair to say that HoloLens is one of the best – if not the best – stereoscopic headset I've tried on, including various generations of Oculus Rift, HTC's Vive, and others.

Anyone who wears prescription glasses and who has tried a VR or AR headset has probably experienced the same thing: either having to take their glasses off in order to wear the system, and hope in the process that there are sufficient adjustment options to tailor it to somewhere close to what you can see clearly, or wearing your regular eyeglasses as well but having them squeezed uncomfortably into the bridge of your nose.

Rather than a strap around the back of the head, HoloLens takes a more complex approach to fit and comfort, though it's a welcome difference. The main body of the headset isn't actually a complete loop – the sides are a little flexible – but inset in the center is a padded hoop which sits on your head. That slides forward and back, as well as tilting and expanding with a knob at the back; you fit it much as you would a baseball cap, and then hinge the HoloLens eyepiece down.

That means very little pressure on your nose, and even if you're wearing glasses you can generally find a workable way to keep them on underneath.

Physical controls and connectors are minimal. Two volume buttons – one concave, one convex, so that your fingers can identify them more readily – are on top of the left arm, while at the back there's a row of five white LEDs and a button that together show system status, battery, and control power/standby.

Inside, there's WiFi 802.11ac and Bluetooth, and a microUSB port is at the back, too, used for charging and connecting HoloLens to a PC for development.

It's that which my time with HoloLens centered around at BUILD this week. Microsoft had set up a temporary "HoloLens Academy" and was offering four-hour introductions to creating apps for the headset; it pared that down to a 90 minute deep-dive for media curious about the same (but perhaps, like me, lacking in the same hands-on coding knowledge).

As Microsoft is keen to highlight, there won't be a need for special apps, or "Holoware", because HoloLens runs the same Windows universal apps as any other Windows 10 computer.

Sure, those with the bare minimum of effort extended will be little more than interactive posters on your virtual walls, but even that – like blown-up iPhone apps on an iPad – will be functional. Meanwhile, any HoloLens app will run on a regular Windows 10 machine.

Development is done using Unity and Visual Studio, and for "My First HoloLens App" I pieced together a simple physics demonstration Microsoft dubbed Project Origami. Two different 3D balls suspended over a virtual notepad surface, with a few folded paper planes and cubes scattered across for the balls to bounce off. Straightforward, but enough to show off each of the key elements of the HoloLens experience.

First, there's gesture control. Given that augmented reality is going to be new to most, Microsoft is pushing a baseline set of simple gestures which will be familiar no matter the app. Central to the "Natural User Interface" (NUI) is the finger-pinch: hold your hand out, forefinger extended, and then pinch it closed to your thumb. It's the HoloLens-equivalent of a mouse click.

Windows Holographic uses gaze to figure out what it is you're actually clicking on, though in reality it's more head-tracking than what your eyes are doing. There's no actual gaze tracking: instead, a central pointer – Microsoft is pushing a simple circle that tracks across the 3D environment, a little like the eyes of the Mysterons from Gerry Anderson's Captain Scarlet – follows wherever it is your head is facing.

Finally, there's voice control, tied into Windows 10's standard recognizer engine. In fact, there are two options for developers: one always-listening, such as the "reset scene" command which, no matter where I was looking, always took the 3D installation back to its initial state, and another that's object-specific. So, when I centered the circular pointer over one of the origami balls, and spoke that object-specific command – I picked "Make it so," because Star Trek seemed appropriate – the ball would drop. Alternatively, I could do the finger-pinch and have the same result.

Using a few straightforward variables and scripts, any Windows Universal app can be upgraded to support Windows Holographic use. Fundamentally that means setting a default virtual position for the holograms in the room, and requesting current head position from the HoloLens hardware.

With preconfigured graphics and scripts, actually piecing together Project Origami was basically a matter of drags and clicks. Clearly, for a HoloLens app you built from scratch, the time invested in crafting attractive 3D graphics and more complex functionality would take considerably longer. Still, even from a short and relatively limited experience of what HoloLens does, it seems like the effort would be worth it.

Your view of the room through the visor is slightly tinted, but it's only when you turn your head and suddenly see a virtual object floating in space that the potential for Windows Holographic hits you. The Project Origami playing field I had fumbled together was clearly a digital creation, but it was also one that seemed embedded into the real world, neither too bright nor too dark. You can move around it, seeing each side and angle, and it stays locked in place.

Things get really interesting when the virtual starts interacting with the real, however. Fixing one of the 3D balls in my gaze, I muttered "make it so" – you can speak at sub-conversational levels and HoloLens will still hear you, even if you're in a room full of other people each chatting with their own headsets – and after a moment's pause, it dropped, bounced off the virtual pad, and fell to the floor.

Actually, though, it didn't fall straight down. Instead, it "landed" on the physical coffee table underneath the HoloLens installation, "rolled" off that, and then finally came to a rest on the carpet. The temptation to put my hand out and try to catch the second ball as it fell was irresistible, but the current build of the HoloLens software has a minimum range roughly equivalent to arm's length: any closer and you start to go "through" the 3D objects.

Temporarily switching on the wireframe mapping shows just what HoloLens is seeing. The world around you for a roughly 5-6 foot radius is a riot of triangles, a Tron-like grid as the headset maps in real-time what physical things the virtual objects could be fixed to, bounce off, or generally interact with.

It's surprisingly precise, too. I moved the Project Origami board to above a workstation – finger-clicking on the 3D object, then moving my head to place it elsewhere – and dropped one of the 3D balls, which promptly rolled into the gap between the keyboard and a USB cable. Crouching down, I could reach out and lift the wire, allowing the ball to tumble out (and promptly get lost down the back of the table).

You quickly get comfortable with the idea of the real and the virtual co-existing. In fact, the urge to reach out and grab a HoloLens object is strong; you have to constantly reset your expectations of just how interactive the whole thing is (a limit primarily on the part of the developer, mind, not HoloLens itself). The spacial audio helps, too: two small red speakers, positioned above each ear, play ambient sound that's linked to the 3D object. As you get closer, the audio is louder; move further away, and it gets quieter.

A final "Easter egg" saw a hole in the floor appear, a jagged-edged gap through which – if you crouched down – you could see a whole origami world. With the carpet fully occluded by the 3D graphic, my brain really was fooled into a sense of depth: like I was actually seeing through into an underworld. Whether it would work so convincingly in brighter conditions is a lingering question, and one which Microsoft's HoloLens team wouldn't answer for me.

There are arguments about whether HoloLens really does create holograms, or if Microsoft has usurped the term when in fact the digital phantoms are something else entirely. Personally, though I can understand the complaints, I think augmented reality is at a stage where it needs all the help it can get to communicate its value to potential users. If that involves a little mislabeling, then I think I could live with it.

Meanwhile there'll inevitably be comparisons drawn between HoloLens and Google Glass, though in reality the headsets are very different. Glass was an all-the-time companion, a constantly-worn system for piping notifications into the periphery of your vision. Although it was at various times confused for augmented reality, in fact it was nothing of the sort.

My biggest concern around HoloLens right now is expectations. With so little hard information about the hardware and software out there, most people only have Microsoft's ambitious demo videos to understand what the experience of Windows Holographic is like. As I found from the prototype, though, the reality is quite different.

We saw what could happen when product doesn't match initial promise with Glass, and Google itself has conceded that unrealistic expectations it itself provoked were instrumental in the first-generation wearable's stumbling. For HoloLens, that disparity between real and promotion is primarily down to the field of view of the displays.

Exactly what goes into the layered transparent eyepieces hasn't been confirmed – we're not sure at this stage who Microsoft's supplier is – but what's clear is that the all-surrounding virtuality of Microsoft's videos for HoloLens is far from the reality of the current prototype. In fact, you're looking at 3D objects through a rectangle in the center of your vision; no field of view specifics have been shared, but it was roughly the same as looking at a 60-inch TV from perhaps 6-8 feet away.

It means that Windows Holographic lives in a fairly small portion of the real-world, and the limits are exacerbated by the gaze control system. Studying my virtual origami world placed on a physical coffee table from a few feet away, when I moved my head up to center the pointer on one of the 3D balls, the bottom edge of the graphic disappeared.

Microsoft's eye-catching demonstration during the BUILD keynote on day one, of a HoloLens wearer surrounded by virtual apps and screens in his living room, is frankly misleading: right now, you could see one window, directly ahead of you, but anything off by more than a few degrees to the side is invisible until you physically move your head.

The degree to which Microsoft can address all this remains to be seen. This is, after all, the first time the standalone HoloLens prototypes have been brought out into the wild, and it's unclear how much larger the active display area in each eyepiece could be expanded in future iterations.

Make no mistake, though, HoloLens is an astonishing creation. Virtual reality is, in practice, fairly straightforward as long as you have accurate-enough head tracking and a sufficiently high-resolution display: all you're doing is blocking out the real and fooling the brain with a digital alternative.

Augmented reality is far, far harder to do well. Rather than replacing the real-world, AR objects must co-exist within it. Any lag, or positional drift, or 3D creations too translucent will all undermine the experience. HoloLens certainly appears to have got it right; in contrast, though making better hardware isn't necessarily "easy", it's a challenge that nonetheless seems achievable.

Will we all be wearing HoloLens headsets in a few years from now? Not all the time; Microsoft has sensibly positioned the wearable as relatively task-specific – donned for entertainment, or communications, or education, or for business tasks like a group of architects exploring the same virtual building – rather than trying to convince us we need to walk around looking like Doctor Who extras 24/7. Plenty will hinge on pricing, too, still as much a mystery as what hardware Windows Holographic runs on.

As I fumbled my way through Unity, piecing together Project Origami, I was reminded of another Microsoft concept from years back, the double-screen Courier tablet that proved so compelling in videos but was axed before it ever reached the market. Back in 2006, well before it ventured into computing hardware with Surface, I longed for Microsoft to take the risk and release Courier. I find myself feeling a similar way now with HoloLens, only this time it seems Microsoft may actually deliver.