Why using mixed-reality video is the smart way to promote your VR game
It's the rare game developer that doesn't want their creation to be a massive hit. While the world of virtual reality (VR) gaming may not yet have seen its crossover, blockbuster moment, there is no shortage of fierce competition for a piece of the pie in a fledgling (for now at least) market of VR hardware owners.
A quick search on the Steam store revealed more than 2000 different VR-enabled apps and games for the PC, the vast majority of which have appeared since 2016. In this context, you want your VR game to grab the eyeballs of potential players, and not let go. Mixed-reality video is a great way to do just that.
Green screen mixed-reality video for VR games is a 2D video production technique that gives the viewer a third-person view of the player in the game's world. Using a hardware and software stack that is explored here, video creators can record or stream live gameplay of the player immersed in the virtual environment, taking the viewing experience far beyond the limitations of the first-person headset view. It's something every VR developer should try.
"Wow, that's really cool!" said pretty much everyone.
HTC and Valve* were among the first to use mixed reality to show off VR in their Vive demo video released in early 2016, and the technique has been used to great effect in trailers for Job Simulator from Owlchemy Labs, and Fantastic Contraption from Northway Games. The Serious Sam developers at Croteam have also embraced it-from their early demos at EGX in 2016, to implementing proprietary mixed-reality video tools in all of their VR titles.
Josh Bancroft and Jerry Makare at Intel® Developer Relations have been working with the technique for over a year, rolling out live demos in 2017 at GDC, Computex in Taipei, E3, and, most recently, at the Intel Holiday Showcase in New York. For them, using mixed-reality video to stand out from the VR crowd is a no-brainer. Having heard the reactions first hand-mostly variations on the subheading above-they understand the advantages of mixed reality over the first-person view in giving viewers outside the headset a compelling sense of the VR experience.
Figure 2: Live green screen mixed reality stage demo by the Intel® team at Computex 2017 in Taipei, featuring Rick and Morty: Virtual Rick-ality.
"The greatest advantage of mixed-reality video for VR is the fact that you can give a glimpse of what it's like inside the VR experience," said Josh. "When you're a third party looking at a first-person VR video, there are things that you don't realize are distracting-when you're in VR, your head movements are smoothed out by your brain, but if you're watching in third person, it's really jumpy and distracting." The third-person mixed-reality view neatly eliminates those distractions.
Sharing is Caring
For many gamers, sharing their experiences with friends is primordial, be it online, local multiplayer, or as a spectator around the TV. With VR, the player is sealed off behind a VR headset, with only the jumpy first-person view on screen, making it harder to include others in the experience. Mixed reality can change that.
"Using the third-person view adds a more social aspect, it's more of a shared experience," said Jerry. "People can see the environment and understand what you're reacting to, which can lead to all sorts of fun things, like people backseat driving and giving you hints while you're playing."
Figure 3. An example of how mixed reality can make VR more social, from the 2016 HTC Vive* video.
Good Business
"The VR market is very crowded right now, and it's hard to distinguish yourself," said Josh. "It's really new and everyone's trying it, but no one has really figured out the secret, smash-hit formula, so there's a lot of opportunity." Enabling your game for mixed reality is one of the ways developers can seize that opportunity.
Firstly, it allows you to create trailers, videos, and live demos with an irresistibly immersive view of the experience from inside the game world, as opposed to using only the standard, first-person view. You can then use these trailers and videos to showcase the game at events, and beyond. "Using that third-person mixed-reality view could give you the edge that gets more people watching your trailer and buying your game," said Josh.
Secondly, adding mixed-reality support to your game throws open the doors to content creators, streamers and YouTubers, allowing them to create their own mixed-reality videos of your game. YouTubers, including Get Good Gaming and DashieGames, have already used mixed reality to great effect in their videos. As the top comment on this DashieGames video puts it, "I could watch this kind of gameplay for hours and not get bored."
Figure 4: YouTuber Get Good Gaming playing VR gladiator title Gorn in mixed reality, while dressed for the occasion.
Many developers are already implementing features designed to help streamers and content creators-such as optimizing for Twitch* streaming, and integrating audience interaction tools. When it comes to VR games, mixed-reality enablement needs to be high on that list. Northway Games built a whole suite of content-creator features into Fantastic Contraption, including mixed-reality support. They wanted to use it in their own streams but, more importantly, they wanted their players to do the same, with the goal of reaching a bigger audience with what they knew was compelling content.
Figure 5. One of Northway Games' live mixed-reality streams, featuring their VR game Fantastic Contraption.
"There is a lot of growth in building integration for streamers, YouTubers, and content creators, and adding mixed-reality support to your game aligns directly with that," said Josh. "If you show that your game will let streamers and content creators produce this amazing mixed-reality video for their audience, that can be really appealing, and ultimately help multiply your reach."
Serious Commitment
For the last couple of years, independent developers Croteam have been on a mission to bring VR to their entire catalog of recent games, including four Serious Sam games, and The Talos Principle. Innovators to the core, it was inevitable that they would explore the possibilities of mixed reality as part of their VR push. "All the cool-looking videos had mixed reality in them, and it looked like the best way of trying to explain to people what VR feels like," said Goran Adrinek, senior programmer, and in-house mixed-reality expert, at Croteam.
The chance to flex their mixed-reality muscles came when they had the opportunity to present Serious Sam VR: The Last Hope* at EGX 2016 in the United Kingdom. "Our plan was to get some kind of real-time mixed-reality implementation going so we could look cool on the stand, capture some promo videos, and also show off the physical minigun controller," said Goran.
Figure 6. The Serious Sam* minigun finding its way into green screen mixed reality at Croteam.
Zero to Hero
However, there was a problem: They had two weeks until showtime, and zero mixed reality enabled in the game. "At that time, we were just discovering what mixed reality was all about, and we tried to do it the same way as the others did: Rendering several views from the game that can be blended together in open broadcaster software," said Goran.
Ever the perfectionists, they weren't satisfied with the results, so they decided to build their own mixed-reality tools in their proprietary Serious Engine that they've been iterating on for the best part of two decades. They chose to bring the entire mixed-reality process into the engine, including capturing the live video, and compositing it in real time inside the game.
It was a bold move that paid off. "We made it in time for EGX 2016, and it worked out nicely," said Goran. "Watching someone play in mixed reality is a much nicer experience than watching a first-person video, and you could see it was grabbing a lot of attention on the show floor."
Figure 7. Croteam's live green screen mixed-reality demo of Serious Sam VR: The Last Hope* at EGX 2016.
Home Advantage
As well as the advantage of needing only one piece of software (the VR game) to manage the whole process, creating mixed-reality video in-game has the added bonus of making the effects look better, because they're mixed with the player's image in the game engine where they're generated.
Coding everything in their engine-from video capture and virtual in-game third-person camera, to green screen chroma key, and compositing-is clearly no small task, but it didn't seem to faze Croteam. "It's hard to tell exactly how much work it was, but not that much, since we didn't have a lot of time to do it," said Goran.
The rapid-fire work in time for EGX was just the beginning of mixed-reality support for Croteam. "After EGX, we continued developing mixed reality until we solved all the problems that were bugging us-like camera calibration, controller delay, and moving-camera scenarios," said Goran. "We ended up with an easy setup process with an in-game setup wizard that anyone could follow."
Figure 8: The Mixed Reality Setup Wizard is included with all Croteam's VR titles.
There is more in store, as they continue to hone their technique and tools. "We did some experiments using the depth information to get even better visual results when mixing the player's image in the game, and this is definitely the way to go," said Goran. "We hope to expand on this approach in the future."
The Mixed Difference
Echoing the thoughts of Josh and Jerry, Croteam saw the clear benefits of mixed-reality support beyond the show floor and their own trailers. "It also helps players easily make videos that can be grabbed directly from the game, and streamed or uploaded to online services like Twitch or YouTube*," said Goran. "We expect a lot more player-generated material to hit the Internet as a direct result of how easy it is to do mixed reality in our games."
Despite the successful two-week hustle pre-EGX, Goran doesn't want to make it all sound too easy. "It is a lot of work, whether you mix stuff in post-production, or if you invest in technology that streamlines the entire process as we did," he said. But, in the end, it's an investment worth making. "The approach and tools are really open for everyone, and even in its basic form, it can produce great results," continued Goran.
Croteam have documented their journey with mixed-reality implementation on their blog-essential reading for any developer looking to explore the technique.
Beyond Gaming
It's not only game developers whose businesses can benefit from the creative applications of mixed reality. One area that Josh highlighted is real estate. As a quick Google search demonstrates, virtual property walkthroughs are already very much a thing, allowing a prospective buyer to visit a property from the comfort of their own home, or a realtor's office. "To take that further, you could make a mixed-reality view of the realtor walking them through the property," suggested Josh. "Or they can be given a mixed-reality video where they see themselves walking around it, so they become more attached to it."
Figure 9. CNN Money* report entitled "Virtual reality is the new open house," about the use of VR in real estate.
From his perspective as a video creator, Jerry can envisage mixed reality being used as a powerful tool in visual storytelling, potentially linked to the immersive theater experiences that have become popular in London, New York, and elsewhere. "There is an opportunity for people to create rich, curated, mixed-reality experiences based on VR," said Jerry. "It's all about adding to the social environments, broadening the reach of what's happening, so people can better understand what's going on and share the experiences together."
In the wake of the Spider-Man: Homecoming – Virtual Reality Experience game released in June 2017, Josh expects to see more commercial brands creating VR and mixed-reality experiences as a promotional tool. "Imagine setting up an environment in a movie theater or theme park where you can insert yourself into your favorite movie or game experience, and hang out with Iron Man, or swing with Spider-Man," said Josh. "Not only do you get to experience that in the VR headset but, with mixed reality, you get a video of yourself doing it that you can share."
Figure 10: The Spider-Man: Homecoming – Virtual Reality Experience* trailer from Sony Pictures Entertainment.
Take the Plunge
But, before we start posting mixed-reality videos of our head-to-head encounters with super heroes to social media, there's more work to be done with mixed-reality video in the world of games, and more creativity to come from the minds of developers.
One such creative leap that impressed Josh was the live demo of the Circle of Saviors* VR game at Tokyo Game Show 2016. The game involves slashing away at dragons with a great big sword, which requires a healthy suspension of disbelief. To complete the effect, the green screen demo was performed by a player in full cosplay, including flowing red cape, so that the final composited mixed-reality video showed the player, in character, immersed in the game universe.
"It was really spectacular," said Josh. "The developer saw the capability of mixed reality, and realized it could be used to do something different, taking the creative vision in a new direction that wasn't possible before."
Figure 11. Cosplayer demonstrating Circle of Saviors, in mixed-reality, at Tokyo Game Show 2016.
And that's the key-showing people what's possible in VR, whether or not they're in the headset themselves. Right now, mixed-reality video does a great job of communicating the idea of "this could be you" to viewers, and VR developers owe it to themselves, and their games, to check out what it can do.
"Showing the player fully immersed in a 3D world using mixed reality is the best option we've come up with for promoting our VR games in 2D media," said Croteam's Goran. "You should definitely try it."
While there are certainly technical challenges involved, they're far from insurmountable, as the experiences of Croteam, and others, prove. Plus, there is an ever-growing number of resources available to help you. "It's not as hard as it seems, and it's strangely addictive," said Josh.
"Once you are able to bring a camera into VR with you, and show that third-person perspective on your virtual world, it will spark your creativity as a developer, and you'll start thinking of cool ways to use it," continued Josh. "It's worth it!"
More stories, tutorials, case studies, and other related material are planned around VR mixed-reality technologies. To stay up-to-date with all the latest news, join the Intel® developer program at: https://software.intel.com/gamedev.
Click here to Join the Intel® Game Dev program for free tools, resources, and opportunities to help you bring the best game experience to the biggest worldwide audience
Source: https://software.intel.com/en-us/articles/stand-out-from-the-virtual-reality-crowd-with-mixed-reality-video