Canon MREAL Mixed Reality Hands-On

Augmented reality is going to be big, and Canon is jumping straight in with its MREAL System for Mixed Reality, a combination of a clever head-mounted display and integration with 3D graphics software to create a real-time virtual world you can interact with. Better known for its cameras, Canon is bringing that knowledge of lenses and optics to an innovative display system that blends the real world with computer graphics, using marker recognition so that physical objects can be picked out and manipulated in the digital environment. We caught up with Canon to try MREAL out, and see what you get for $125,000.

Canon is implemented true mediated reality with its headset, using stereoscopic cameras on the front of the HM-A1 HMD to take a view of the real world and then overlaying computer graphics, which are then fed to the twin displays inside. They run at 1280 x 960 resolution, higher than many cheaper HMDs we've seen, and use a specially created free-form prism display system that Canon claims means less distortion and fewer optical aberrations around the periphery.

Capturing and displaying graphics is only part of the overall system, however. The MREAL setup also requires tracking data, to know where the user is looking and what they're interacting with, and of course a software platform that hooks into whatever 3D environment you're exploring. On the sensor side, there's a combination of visual markers and an optional gyroscopic sensor, the latter of which tracks the orientation of the headset in space per frame of video.

It's the markers, though, that are more commonly relied upon. MREAL can work with optical and magnetic sensors, including third-party brands, but the most obvious are a series of QR-like glyphs which can be used to mark the sides of a physical mock-up. By tracking those glyphs, MREAL can map the movement of the virtual object against the user's manipulation of its physical counterpart.

So, a basic model of a car, or a camera, or the control surface in the plane could be marked out with a few MREAL glyphs, and then Canon's system will overlay whatever proposed controls or components are intended. The wearer can interact with those controls as if they were real, able to "look around" the environment with the physical perfectly matched to the digital representation.

On the software side, Canon's MR Platform has been designed to hook into the most common CAD and 3D visualization packages, with an MR Platform SDK to make integrating existing graphic design software with MREAL straightforward. RTT, the makers of 3D visualization software RTT DeltaGen, has already confirmed it will be combining the package with MREAL; existing customers include Audi, BMW, GM, Ferrari, and Porsche, as well as Adidas and Electrolux.

It's not just prototyping and industrial design that Canon sees as benefiting from MREAL, however. Another avenue the company is pushing is museum use, where visitors would be able to don an HM-A1 headset and have characters – whether famous faces from history, dinosaurs, wild animals, or fantasy constructions – appear in the real-world with them, interacting as if themselves real. Another possibility is retail, where designers of custom furniture, architects, and others could use MREAL to walk their customers through the design refinement process, tailoring the final product to them without costly iterative prototyping along the way.

Compared to the wearable displays we've seen on Google's Glass and from others, Canon's is obviously a league ahead. The blending of the digital graphics and what's actually going on around you is surprisingly good, with the mapping of the two very accurate. You can see your hands – if the system decides they're relevant to the environment – and after a little acclimatization, it really is like you're handling a more complex object or coming face to face with a dinosaur.

We tried several demos with the MREAL system, tailored to different potential clients. In an educational-style environment, we were able to interact with a virtual dinosaur, walk around it and pet it, while still seeing other people in the vicinity not wearing a Canon HMD. In another setup, we were able to virtually "kick the tires" of a digital car, customizing colors and trim, exploring the dashboard, and generally seeing what we could expect to be driving despite the real-world object being far more basic in its design. A virtual earth-mover allowed us to experience the cabin and identify ergonomic flaws, such as controls that blocked access to the cup holder, and which might normally demand a physical prototype be built before the issue was identified. There's more details in our hands-on videos.

As Canon and its partners see it, the big difference between something like Glass and its MREAL system is the introduction of haptics. "We're already overloading the eyes as it is" Simiosys' Christopher Stapleton explained to us. "The question is what's going through what channel, and how. So, this whole aspect of multitasking isn't about the number of tasks, it's about competition for attention."

"You have all the senses, in all dimensions and all directions, all this impact, we're overloading the eyes in proportion to the hearing, or the touch. The eyes have depth-perception to a certain extent, but the only interactive sense we have is touch, so the aspect of mixed-reality and haptics is a huge jump in how much we can do. What [Google] is doing is too much in one area."

Meanwhile, the traditional idea of a GUI – whether text or icons – is quickly becoming overwhelmed by the complexity of what today's systems can deliver. Instead, Stapleton argues, systems like MREAL can take what would be a very complex interface and make it more naturalistic. Today's users aren't looking for ever-increasing menus and toolbars, he says, but gesture response, social interaction, and a more human way of encountering the digital world.

Perhaps unsurprisingly, all this doesn't come cheap. Canon will be selling the entire MREAL Mixed Reality System – the HM-A1 headset and the MR Platform software – for $125,000 from March 1, with a $25,000 annual maintenance fee on top. That might sound expensive, but Canon balances it against the existing costs companies face of producing a hardware prototype. That process – which can be relatively quickly and affordably mocked-up virtually using MREAL – is an even more expensive one, and it's easy to see how MREAL could earn its keep over time.