Virtual Interfaces Finally a Reality: What Web and User Experience Designers Need to Know about Virtual Reality (VR) and Augmented Reality (AR)

The virtual reality (VR) and augmented reality (AR) spaces are on fire as of late. Just look at the news. Nearly every day I’ve come across at least a couple of articles discussing the virtues of VR in my Twitter feed - and we’re only getting started.

As a web and user experience designer, I knew I had a responsibility to understand this area, but as a non-gamer who has been out of the 3D graphics space for a long while, I didn’t know where to start.

With the news coverage, product releases and even the possibility of our agency, IdeaBase (part of Kent State University), entering the VR market, this is something we’re all going to need to explore and discuss with clients sooner than later. Here’s a few things I’ve learned about virtual and augmented reality in the last few weeks. Hopefully, this will be helpful to other professionals as well and hey, it might even get you excited about the future!

THE STATE OF USER INTERFACES

It’s no secret that the traditional user interface market is crowded and perhaps slowing in innovation. It’s becoming clear that Apple and Google have won the still-booming mobile operating system platform war. Don’t get me wrong - the smartphone market is still strong and interesting. By 2020, 70% of people in the world are expected to own smartphones after all. However, there’s a great deal of chatter in the technology and communication industries about what’s next.

Voice control has really taken off over the past few years, led this year by the unlikely contender, Amazon Echo, and the desktop and mobile platforms have their own integrated voice control solutions as well. Smartwatches, last year’s big story, are also gaining in popularity (over 50% of smartwatches sold are from Apple, with the rest of the industry fragmented among the other ). Smartwatches, and increasingly voice-controlled interfaces are really just an extension of mobile and desktop operating systems.

Is there space for a new user interface paradigm where the experience mirrors reality, with completely new interactions?

VR AND AR DEFINED

That brings us to VR and AR. By definition, virtual reality refers to fully immersive software where you’re completely surrounded by virtual, interactive environments. Augmented reality refers to technology that mixes the real-world with interfaces and objects that modify the real world. Both types of technology have reached an all-time high in the industry lately, and, while the initial solutions are very interesting, experts expect that future solutions will be a combination of each.

A LITTLE HISTORY

Revenue from virtual and augmented reality is now estimated to reach $150 billion by the year 2020, but if VR sounds familiar, that’s because it’s not a new technology. That’s right - it’s been around since the 1950s and 60s.

A filmmaker created the Sensorama, a system for augmenting experiences in movies, in 1962, and a head-mounted system that had to be suspended from the ceiling was proposed in 1963. An oral history of virtual reality is documented the Verge’s excellent The State of Virtual Reality article, which I’ve referenced many times in this post.

VR excitement continued throughout the late 80s and early 90s, with technologies like a wired glove (VPL’s DataGlove) and the Nintendo Power Glove in 1989, which help track the wearer’s movement and location.

In the mid 90’s, though, the personal computer industry was coming to fruition, and with the switch from command line interfaces to visual interfaces, enthusiasm around VR began to wane. There were simply more opportunities and less barriers designing within Microsoft and Apple’s popular graphical user interface systems than continuing on the VR path.

Then, in the late 90’s, the innovation and excitement turned toward the Internet in the late 90’s, with virtual reality research primarily relegated to military and academic research – never reaching wide consumer popularity.

NEW CONTENDERS

As the personal computing industry matured and computers became more powerful, access to information became instantly accessible via the internet and later through the transition to mobile devices. This allowed a few consumer-oriented devices began to emerge.

Google Glass, a project that has since been discontinued, exemplified a potentially useful and accessible applicable for augmented reality. Glass was publicly announced in April 2012 and was made available to developers for $1,500. While not a commercial success, Google Glass showed wide interest in the space, and provided a proof of concept for future solutions (including Google Cardboard and Microsoft HoloLens).

Perhaps most famously, Palmer Luckey, who worked as part of the design team at University of Southern California’s Institute for Creative Technologies Mixed Reality Lab, raised money on Kickstarter and created his own virtual reality solution that he named the “Rift”. The Kickstarter campaign raised $2.4 million and his company, Oculus VR, was acquired by Facebook in 2014 for $2 billion.

Virtual reality devices like the Rift prioritize greatly advanced technology that was previously too cost-prohibitive and powerful to be integrated into the earlier VR solutions:

Two high resolution OLED panels, one for each eye with a faster refresh rate create a visual experience that appears more natural and realistic to its wearers.
External tracking sensors and motion controllers allow the system to respond to body movements and movements within the 3D space of a room.
Modern headsets include high-quality audio that further increases the user’s’ sense of presence in a virtual space.

Interestingly, high-end VR and AR devices remain tethered to traditional desktop computers (which cost about $1,000 at the high end) or game consoles, while lower-end solutions require the use of a high-end smartphone, making VR (for now) dependent on other devices for full functionality.

THE DEVICE LANDSCAPE

While the Oculus Rift seems to be the most widely covered solution, it’s definitely not the only contender, and it’s anyone’s guess which device (if any, including unannounced solutions) will be the successful consumer product. Until now, Oculus and HTC were only shipping pre-release, developer versions of their devices, but many consumer-ready devices are now reaching consumers.

Low-cost (perhaps inferior) solutions like Google Cardboard have already been in use. Google says more than five million Cardboard headsets have been shipped to date. Many, though, were given away by free from companies like the New York Times to support their own VR initiatives.

In 2016 alone, $1.1 billion has been invested into the virtual reality industry and Goldman Sachs has predicted that millions of headsets will be in people’s homes by the end of the year. It’s safe to assume innovation will continue and new devices will emerge throughout the year.

At the present there are some interesting solutions that vary greatly in price, functionality, features and availability (this list is by no means comprehensive).

VR Solutions on the Market

Device	Cost	Includes	Add-ons	Description	Availability
Samsung Gear VR, a partner with Oculus	$99	Back band, top band, window cover	Requires Samsung Galaxy S7, Galaxy S7 edge, Galaxy Note5, Galaxy S6 edge+, Galaxy S6, Galaxy S6 edge ($600 - $700) Gear 360 (3D Capture - Spring 2016)	Watch movies in your own private cinema, be at the center of your favorite games, and socialize with friends in a virtual world all your own. The lightweight design and intuitive touchpad drop you right in the action and allow for hours of immersive entertainment.	Now
Oculus Rift (Facebook)	$599	Includes a headset, sensor, remote, cables, Xbox controller, and Lucky’s Tale.	Handheld controllers (price TBA) Requires Desktop Computer (average $1000)	Rift uses state of the art displays and optics designed specifically for VR. Its high refresh rate and low-persistence display work together with its custom optics system to provide incredible visual fidelity and an immersive, wide field of view.	August 2016 (Developer version available now)
HTC Vive	$799	Headset, two wireless controllers, and two base stations enabling 360° room-scale motion-tracking. Plus, you’ll get Tilt Brush, Fantastic Contraption and Job Simulator free	Requires Desktop Computer (average $1000)	Uniting passion, talent, and innovation, Vive delivers on the promise of VR with best-in-class technology and content. 360˚ motion tracking means realistic room-scale experiences.	May 2016
Google Cardboard	$20 - $40	Google Cardboard viewer	Any modern smartphone ($400 - $600)	Get it, fold it and look inside to enter the world of Cardboard. It’s a VR experience starting with a simple viewer anyone can build or buy. Once you have it, you can explore a variety of apps that unfold all around you.	Now
Playstation VR (formerly Project Morpheus)	$399	VR headset × 1 Processor unit × 1 VR headset connection cable × 1 HDMI cable × 1 USB cable × 1 Stereo headphones × 1 (with a complete set of earpiece) AC power cord × 1 AC adaptor × 1	Sony Playstation IV Console with PlayStation Camera ($450)	Discover a new world of unexpected gaming experiences with PlayStation®VR. Redefine your expectations of immersion in gaming with moments so intense your intuition takes over. Step into incredible virtual worlds and overcome new challenges in extraordinary ways.	October 2016 (Pre order Summer 2016)
Microsoft HoloLens	$3000 Developer Kit	Microsoft HoloLens Development Edition, Clicker, Carrying case, Microfiber cloth, Charger, Micro-USB 2.0 cable	Windows 10 PC able to run Visual Studio 2015 and Unity 5.4 ($1000)	Microsoft HoloLens is the first fully untethered, holographic computer, enabling you to interact with high‑definition holograms in your world.	Now (Developer Edition only)
Zeiss VR One	$129	>td >Supports smartphones with display sizes between 4.7 and 5.2 inches ($600-$700)	It's finally possible to experience virtual or augmented reality any time, any place thanks to the new ZEISS VR ONE! VR ONE is a mobile Virtual Reality (VR) headset that works with smartphones.	Now
Leap Motion	$80	Leap Motion Controller, 2-piece flat VR Developer Mount for the Leap Motion Controller (compatible with the Oculus Rift CV1, HTC Vive, and Oculus Rift/HTC Vive Development Kits), 15' USB cable extender, Custom 3M adhesive, Getting started guide	Windows PC with NVIDIA GTX 970 / AMD R9 290 equivalent or greater with compatible HDMI 1.3 video output ($1000)	Reach into new worlds with Leap Motion technology. Attach a Leap Motion Controller to the front of your virtual reality headset using the VR Developer Mount, and bring your hands into VR using Leap Motion’s Orion software.	April 2016 (VR version)

The two high-end solutions, HTC’s Vive and Oculus Rift, were recently reviewed by publications like Engadget, which focused on the differences in comfort, features and price between the two. Engadget concluded that the Vive “simply has more “wow” factor when it comes to showing off the benefits of virtual reality,” while the Rift offers more ergonomic comfort, but the race is far from over.

Samsung has partnered with Oculus Rift for its Gear VR solution, which integrates with its Galaxy smartphones. The company announced that everyone who preorders its latest smartphones will be eligible for a free Gear VR headset. The competition is going to be fierce.

APPLICATIONS FOR VR

Technology means nothing if it doesn’t provide some kind of useful application for real people. So, assuming a consumer has purchased a VR solution, what can they actually do with it?

There are a few popular uses emerging, and in order for the market to sustain itself, strong content will continue to be a priority. Many applications, especially for gaming, are paid, as there isn’t a strong contender in the VR marketplace (although the popular Steam software from Valve is now a partner with HTC). Free and web-based software solutions also exist.

Virtual Tours

Virtual tours are made by capturing 360 degree, 3D images from an actual 3D space and place the wearer in an actual environment that he or she can navigate. The interfaces allow users to stare at targets on the screen to navigate around the space. This is invaluable for many industries that concentrate on physical spaces, including realtors and architects. Current high-quality image technology is expensive, with cameras from companies like Matterport starting at $4,000. GoPro has also announced VR capture technologies, including a high-end solution called the Odyssey, which costs $15,000.

Gaming

By far the most common use for virtual reality, many of the VR solutions bundle gaming software, and solutions from Oculus, HTC and Sony are targeting the high-end video game industry. These games include advanced 3D graphics that place the player in the middle of the action, using a combination of movement and audio to create a realistic experience that traditional, screen-based game consoles lack.

Storytelling and Entertainment

Journalists and filmmakers interested in creating emotional, tangible experiences with storytelling have also found virtual reality to be an effective solution. Rather than presenting a linear experience, virtual reality allows users to physically navigate and explore an environment. Disney’s “The Lion King” show offers a 3D virtual experience and Fox Innovation Lab released “The Martian VR Experience” as a companion piece to its “The Martian” movie. Similarly, the sports industry has already started to capture games in 360 degree video. “As journalists, we always seek to help readers understand what life feels like in the places we cover. Virtual reality allows us to do that in an entirely new way”, said New York Times journalists Ben Solomon and Leslye Davis in a recent article from Poynter. The New York Times recently sent its print subscribers Google Cardboard for free in support of its VR initiative.

Advertising

Closely related to the storytelling angle, advertisers also have interest in the VR experience, as these experiences require 100% of the wearer’s attention and, consequently, advertising and marketing opportunities are difficult to ignore. Studies suggest that the average response rate to direct marketing is less than 1%, whereas VR experiences have garnered a more than 15% response rate. Additionally, the 3D environment allows for nearly infinite virtual space on which to place advertising.

Education

Google has already shipped its Cardboard to classrooms with its Google Expedition program, providing educational solutions to K-12 students. Additionally, VR provides opportunities for students and researchers in healthcare and clinical settings to better understand anatomy and psychological effects.

Adult Film Industry

Of course, the adult film industry has jumped on board as well. Pornhub recently launched its own VR porn channel, and comparisons have been made to the adult film’s influence over the VHS victory over Betamax and on Blu-ray over HD-DVD.

Software and UI Design

While most definitely in its infancy, virtual and augmented reality solutions have allowed companies like Microsoft to showcase interactive versions of its Skype software and dozens of smartphone applications already offer augmented reality functionality. A software interface was recently built to navigate the Reddit website and Tilt Brush, recently purchased by Google, is an application that allows users to draw pictures in the air in front of them and interact with their creations - blurring the lines between productivity and entertainment. The popular gaming software program Steam from Valve now allows users to display everything from their computer’s desktop on its VR dashboard. This has the potential for VR to supplement common activities, like messaging and Internet browsing, traditionally thought of as 2D experiences. The opportunity for software and web designers to overlay their applications and interfaces upon the real world is by far the most exciting (but challenging) application for designers.

DESIGNING FOR VR

This brings us to the design possibilities (and constraints) of VR. How can we as web and user experience designers integrate virtual reality into our decision thinking?

Much like the transitions from print to audio and video and desktop computing to mobile, many of the existing design strategies must be changed or rethought in order to accommodate these new interfaces. That said, I’m confident that the principles of human computer interaction, including appropriate affordances, signifiers and feedback, will continue to serve us well here.

While still very much in its infancy, VR has a few design constraints that we’ll need to consider:

Experiences with large amounts of movement can make some people nauseous. They might also be more fatiguing than more passive experiences. We’ll have to consider how we accommodate people who are sensitive to motion, and provide accessible alternatives.
Sony and Oculus have placed age restrictions on their hardware, recommending use only for users older than 12-13. Google recommends age 7 and up, with adult supervision.
Comfort is also a factor. Consider how long it will take for a standing user to become fatigued. Additionally, some of today’s headsets are heavy and may add additional physical strain.
Interfaces based on movement may lack appropriate visual cues and be difficult to determine intent - for example, selecting or interacting with a particular element.
Typical UI elements, such as buttons, may not work in virtual interfaces. How can we display information in view that takes advantage of the 3D environment and allow interaction via the multiple motion and audio sensors available to us? Additionally, text will need to appear larger to maintain readability, while long passages of text may be best left for 2D displays.
With the graphics-rich properties of VR, how can we build for users who are on slow Internet connections or machines that don’t meet the minimum specifications?

The blog at www.twentymilliseconds.com covers may of the interface design principles that do and don’t work in VR.

At the moment, development and design for VR more closely mirrors 3D modeling and game design than traditional graphic design and web development. I would predict that developers and designers with these skills will quickly be in high demand. That leaves people like me in a strange position. While we can consume existing software and perhaps participate in the capture of real 3D environments, VR presents real problems for development and prototyping.

Current prototyping tools only focus on 2D environments and while old standbys like sketching will be useful here, they won’t fully provide a proof-of-concept of these interfaces.

Additionally, development solutions currently present a very high learning curve. There are some interesting developments coming to the web. An experimental JavaScript API called WebVR provides access to virtual reality devices within the browser. Additionally, a Mozilla-led effort called A-Frame was built to target developers who understand HTML and CSS. However, no clear standard exists.

Early solutions will likely attempt to replicate, but not improve upon, the existing web and software experience. As Fast Company notes, “there’s nothing inherently amazing about browsing the same web you know just because you’re inside a virtual reality headset. In fact, it’s inherently worse. The virtual screen is no larger than your real one. The type is actually quite a bit more pixelated than it is on any modern monitor. The experience is a lot like trying the Oculus Netflix app: Fun that it works, but also, inferior to just watching a movie on your laptop or TV.”

STUDYING AND TESTING VIRTUAL EXPERIENCES

Conducting user testing and studying virtual and augmented reality experiences will allow us to better understand what are effective design solutions. We’ll need to test for ergonomic issues as well as cognitive understanding. That means we’ll have to have spaces where the latest hardware is available and researchers and developers who understand these new advancements.

Several labs are prominent in academia with relation to the study of virtual, augmented or mixed reality. They include:

Stanford University’s Virtual Human Interaction Lab (which has studied interesting psychological concepts, including simulating a crowd to help overcome anxiety over public speaking)
University of California’s Mixed Reality Lab (which receives funding from the military)
Rowan University’s Virtual Reality Center
University of Minnesota’s Virtual Reality Design Lab
University of Buffalo’s Virtual Reality Laboratory
University of Texas at Austin’s Virtual Reality Laboratory (also includes the study of haptics)
University of Houston’s Virtual Reality Clinical Research Laboratory (studying anxiety and substance-related disorders through VR)
Texas State’s Virtual Reality Technology Lab

WHAT’S NEXT?

Innovation in this area will be swift, rendering many current technologies (as well as this blog post) obsolete. Like any upcoming technology, we’ll need to develop an understanding of the current landscape, while also not becoming too attached or focused on any specific technology. As Joanna Stern from The Wall Street Journal notes, “all of this hardware is more Apple II than iMac. It’s still the very early days.”

Within the next year, we’ll see these headsets get cheaper and lighter. I would expect as well that we’ll gradually see the devices become untethered from traditional desktops and smartphones, and instead operate as independent units. Software makers will continue to innovate as well, with more options from which to choose, exclusive titles from each hardware maker and easier ways to integrate VR into existing platforms, like the web.

We’ll also see new offerings from companies who are just dipping their toes into the space. Apple Inc., notorious for exercising caution on entering new markets, is rumored to have a secret lab with hundreds of employees working on AR and VR projects. They’ve acquired multiple VR companies and hired some top field experts in the field. Apple CEO Tim Cook described virtual reality as “really cool” and said “it has some interesting applications.”

Video providers like Hulu, Netflix and Amazon are also said to be secretly exploring the VR space. Additionally, companies like Microsoft and Sony have yet to release their previously-announced solutions.

And then there’s Facebook, the social media behemoth who arguably reinvigorated the interest in the technology. The company of course sees great opportunities for virtual reality. As VP of Engineering for Facebook Cory Ondrejka explained to The Verge, VR allows people to “share a sense of place and presence, and really move beyond what we’re used to with what we can share from a cellphone or photo.” He says of Facebook’s acquisition of Oculus allows the company “to partner with a team that’s built this incredible technology, that pushes us across the discontinuity from malfunctioning VR to functioning and amazing VR, and looking at what that platform creates for communication, creativity, expression, it’s right in the middle of our mission.”

As user experience and design practitioners, we need to embrace this change and remember that communication should not be tied to any particular medium. Effective storytelling, well-designed interfaces and attention to user needs should supersede technology. I’m confident that as you’re reading this post via your VR headset or AR browser of the future, user experience will still be a valuable profession. As human beings continue to thrive and interact with the world, so too will user experience designers, constantly improving; constantly evolving.