Steve Mirsky:    Welcome to Scientific American's Science Talk, hosted on December 15, 2015. I'm Steve Mirsky. Virtual reality is coming. It's been coming for a while, but it's getting pretty serious and it's going to be happening in a big way probably in just a few years. To hear how, I'll turn it over now to Scientific American's tech editor, Larry Greenemeier.

Larry Greenemeier:   The recent launch of Samsung's Gear VR promises to be a watershed moment for virtual reality. The headset could be the best thing to ever happen to consumer virtual reality or it could prove once and for all that people do not need to wear a visor to get an immersive experience. One look around you at all of the oblivious people stooped over their smartphones tells you that those devices are already pretty immersive. This attempt is not the first time there's been a push to bring virtual reality out of the lab and into living rooms.

                           The 1990s saw crude VR efforts such as Nintendo's Virtual Boy and Iglasses spelled with an I from a company called Virtual IO, to name a couple. That effort was derailed by technology that was underwhelming to use, expensive to buy, and tended to give people motion sickness. They had movies like The Lawnmower Man and its' sequel failed to help virtual reality's image as well. That's changing.

                           Google and Samsung are now selling virtual reality visors powered by smartphones. By the 2016 holiday season, you should be able to buy visors from Sony, HTC, and Facebook's Oculus division. They won't run on smartphones but they'll still be a lot cheaper than today's high end VR visors which cost tens of thousands of dollars.

                           Before virtual reality takes another run at the mainstream, let's take a look at why the technology was developed in the first place, how it works, and where it's going to get us up to speed on VR's evolution and future, I spoke with Ken Perlin, a New York University computer science and virtual reality pioneer. Here's an edited version of our conversation.

[Beginning of Interview]

                           Virtual reality. What is it?

Ken Perlin:         In its' broadest sense, virtual reality is an attempt to give people a full sensory immersion experience into some other reality other than the literal one that they are in.

Greenemeier:    And what was its' original purpose?

Perlin:                The first people who seriously developed virtual reality were Ivan Sutherland and his student, Bob Sprout, back in 1968. That was almost half a century ago. They were focused on being able to visualize scientific and mathematical objects, and walk around them as though they were part of the experience of the real world. They built a very large device which they nicknamed The Sword of Damocles because it was a very large contraption that hung over your head and carried the headset with it as it moved around on a giant boom arm.

Greenemeier:    So what's happening when someone puts on a virtual reality visor, whether they're looking at a smartphone screen or the screen within the visor. What's happening?

Perlin:                What virtual reality generally does is it maps your head movements to the position of a virtual camera in an alternate world. So it's as though, in a computer graphic alternate reality, your eyes are the camera moving around. The sensation to you is as though you are physically moving around in that space.

Greenemeier:    The early technology, was it using similar sensors – light sensors or accelerometers? Things like that? How was it doing it back then?

Perlin:                The earliest technologies, the ones that date back to 1960, because they were on these giant mechanical boom arms could use the mechanical affordances of essentially robots in reverse. This thing turns mechanically and we can measure that movement and then build a model of where the position and orientation of the person's head must be, and therefore give that person the correct graphics.

Greenemeier:    How has that evolved over time?

Perlin:                Over the years as Moore's Law continues and therefore, computers get progressively faster and less expensive, different techniques have been used. Gradually, the technology started becoming available to do optical tracking and then very recently, to do little solid-state gyroscopes. That was an outgrowth of the fact that those were being developed anyway for smartphones. 

Greenemeier:    How is virtual reality being used today?

Perlin:                Virtual reality is being used for many things. The initial commercial push that's pushing it out to millions of people is computer games, which is the same push that pushed 3-D graphics out to millions of people back starting in the late '90s. However, as we've seen with 3-D graphics, there are many, many other things that you can do with it once commoditization makes it inexpensive enough and therefore widely available enough.

Greenemeier:    What does the software have to do in order to take the information from the sensors and deliver the vision that you want into the visor?

Perlin:                Ideally, as in our research at NYU, you want people to be able to walk around freely as though they're walking around in the real world, which is why we are focusing on free, walk around, untethered solutions. 

Greenemeier:    How is that accomplished?

Perlin:                The way we're doing it now is we're taking a Gear VR, which is a fairly inexpensive device, and we're putting trackers on it and then using optical trackers to detect the position of the Gear VR. We take the information of orientation that's already in the Gear VR, combine that – that's a technique called sensor fusion when you combine different input techniques – with the position data, and thereby allow multiple people to walk around a room just wearing the Gear VR and have a virtual experience together.

Greenemeier:    How does it work when you try to capture the motion of the hands. Outside the visor, what has to be done to be able to represent the hands?

Perlin:                To represent the hands, it depends what you want. If you just want position, then there are fairly straightforward solutions. All of the people who are coming out with these commercial devices have different sorts of untethered trackers. In other words, no wires needed. There is one solution coming out from Valve HTC for their live system, based on their lighthouse trackers. There's another solution coming out from Sony for their Morpheus based on the same eyesight trackers that you see in the PlayStation four. Then there is another infrared optical solution coming out for the forthcoming Facebook Oculus. They're all similar in ways.

                           If what you want is full finger interaction, then optical solutions don't really work because, in fact, your fingers may not be visible. We, for example, have started experimenting with the Perception Neuron, which is a glove that puts little inertial sensors at different places around your fingers.

Greenemeier:    Tell me now about how virtual reality moves, I guess, down the body. You had mentioned being able to include the feet as coming up.

Perlin:                The major commercial releases of virtual reality that are going to be showing up in the first half of 2016 – the three major platforms track your head and they track your two hands. In order to have a full social experience with other people of being in a world together, you also need to know where your feet are. Once you know your head, and your hands, and your feet, then you can build a computer graphic representation of everybody. I can see you in the shared virtual world as we walk around. In order to address this, one of the things that – technologies that has come out of our lab here at NYU is a floor mat that can act a little bit like the Marauder map in Harry Potter. As you walk around on the floor, it detects the pressure of your feet. Those footsteps, together with the hand and head controllers, give you a full body.

Greenemeier:    So that's something you're actually able to work with now?

Perlin:                Yes. That's a technology that has spun out of our lab. There is now a commercial company that is working with the various game providers to create that opportunity. The reason that's interesting is that virtual reality could split into games I play by myself and experiences, or social experiences between people. We find the social experiences between people to be much more interesting. 

Greenemeier:    One of the big goals you have is to re-create the holodeck. Tell me a little bit of that. I'm kind of curious how that's different than the AlloSphere out west and The Cave technology that's sort of been around for a few years? What is your vision? You're sort of getting there, but what is eventually your vision for virtual reality?

Perlin:                The eventual vision is that we will not call it virtual reality. We will call it reality. In ten years, at most, lightweight glasses that anybody can wear without sticking out socially. Within 15 or 20 years, contact lenses. At some point, it will just become normal for people to walk around physically in the world and be able to see and hear whatever they want. That means that you and I can have a social experience. We're talking to each other and as far as we are concerned, the things we want to see just float in the air between us as though they're part of the world itself. That's very different from these large projector-based technologies like the AlloSphere or The Caves from the early '90s. Those technologies did not allow things to happen between you and me in the social context. 

Greenemeier:    So how do you go about shrinking down the technology? You were talking about gloves, and mats, and things that sort of keep you in a certain space. What has to be done to basically untether you from a room?

Perlin:                The nice thing about this technology is that it all advances – this whole realm of connected technology is all advanced together with Moore's Law. As computers get more powerful and less expensive, we can turn more and more to what is now called inside out tracking. The device that I am wearing is able to have a model of the world wherever I am and say, "Oh, there's a table. There's a door. There's another person," but it's all on the wearable device itself. We're seeing the beginnings of this now with Microsoft's HoloLens and Google's Project Tango. It's going to start to take over as computation gets more powerful.

Greenemeier:    Would you still, in the near term, need to have sensors specifically placed on hands and feet? Maybe something you put on your shoelaces or a bracelet?

Perlin:                In the near term, you will rely on wearables. I suspect that as time goes on, you won't need that anymore.

Greenemeier:    What are the most common uses of virtual reality today? I had written stories a few years ago about helping soldiers with post-traumatic stress disorder. There was something else I'd seen about helping people with body image disorders. How is it most commonly used today other than gaming?

Perlin:                Once you and I have the ability to have a conversation with each other in a transformed space – so we could be having this conversation in a café in Venice. We could be inside the space station. We could be looking at a molecule, or an economic chart, or a review of a movie floating between us.

The explosion in possibilities is going to be somewhat like the explosion in possibilities from the very earliest web browsers in 1993 to about 14 years later when the iPhone showed up. In those 14 years, people realized that once you had the web, you could run the whole world in the web. It was everything. It went from, "What is this good for?" to, "How could we ever have lived without this?" I think that as this becomes unobtrusive, and social, and becomes about communication between people, the applications are going to explode to the point where we will just think of this as part of our everyday reality. 

Greenemeier:    Are there any concerns about psychological effects of spending a lot of time in virtual worlds? Is that a concern at all?

Perlin:                I'm not so worried about psychological effects of spending time in virtual worlds. One reason I say that is that every single time a new technology comes out, from books, to television, to movies, to radio – you can go back through the literature and find people who said radio is going to destroy people's ability to deal with reality. The telephone is going to screw people up. Every single time. People are very, very good at understanding what reality is. I think it's one of our great strengths. We know the difference between experiencing something this way and that way. Once something goes into our culture and is absorbed by the culture, we tend to use it appropriately. 

[End of Interview]

[End of Audio]