VR Recording With Mixed Reality: Our First Trials

Today we're going to talk briefly about what we're doing in virtual reality – specifically in attempting to capture what it's like to be IN the virtual realm. It's not enough, we think, to just show the flat feed – though we've had to do that already several times. It's also not enough just to show two screens – flat feed and a real view of the user with a VR headset on. That shows what's going on, certainly, but it doesn't go far enough in showing what the user is seeing, hearing, and experiencing.

One of our earliest inspirations to create the setup we're aiming for comes from developer Dylan Fitterer. This fellow created the video you're about to see – he's developed the app you'll see too: Audioshield. Look at how simple this is. It's still flat, certainly, but it gives you a much better idea of what's going on than if you only saw the game.

Simple. Pretty great for this kind of game.

Here's what we're working with – without any sort of mixed reality:

Besides the appearance of the controls in the frame, there's very little connection between what you're seeing and a true understanding of what it's like to experience the VR. We want to bridge that gap. Bridge it better than we have already, that is.

We've seen some AWESOME examples of what can be done with mixed reality recording. Valve, in particular, has been able to describe the VR experience with great success.

That's awesome.

To make the video you see above a reality, Valve used recording software, the Steam VR plugin for Unity, and a 3rd HTC Vive controller attached to a big ol' camera. The software detects where the camera is thanks to the controller, creating a second camera within the VR game.

You need one camera in the game and one camera in real reality.

The final video is mixed after both videos are captured.

But you DO need developer access to whatever game you're recording, to make this happen. You also have to be able to access the game in Unity, of course.

In the future we're hoping SteamVR will allow this 3rd controller-as-camera setup in all VR apps. For now, we've got to get creative with apps that haven't gone the extra mile to build this functionality in to their apps.

The first app to do this is Fantastic Contraption.

Look at this trailer.

Super!

We can't do this yet with any other app we don't have developer access to. Yet. Until then, we CAN try other things. Recording with a green screen – all sorts of things.

Our first trial is ROUGH. This is just what happened when we got started. It wasn't really meant to work especially well – as such, you might get dizzy whilst watching.

You still need to give your mind a stretch to connect what the user is doing and what you're seeing. That's not what we want.

Our next trial takes a first-person perspective, recording with a camera attached directly to the front of the HTC Vive. Granted, there IS a camera there already, but we wanted a BIT higher quality result than what would result from just that.

The game you see above is a Beta release of Cyberpong VR – we recommend you check it out as soon as possible (on Steam right now).

Still very rough!

But we're getting closer.

Next we're going to be backing up a bit more, using the headset as a camera (without wearing it), and a few other odd ways and means.

Are you attempting to capture VR in similar ways? Let us know what you're up to! Contact me on Twitter (@t_chrisburns) to have a chat!