In this series of interviews. I ask scientists, engineers, and ethicists how technology might change our future. We had these conversations during the research for my book, Welcome to the Future (Quarto, 2021).
Interview 7 – Toshi Anders Hoo, media expert
Toshi Anders Hoo directs the Emerging Media Lab at the Institute for the Future in Palo Alto, California. His group researches the impact new media such as VR and AR have on society, now and into the future. He has also worked with the famous futurist Ray Kurzweil. I first spoke with him in September 2019. We then had another conversation in December 2020.
When you think about the future of VR and AR, what seems most exciting to you?
To me, what’s most exciting about the future of virtual reality and augmented reality is seeing these as communication and collaboration platforms.
A lot of people talk about the future of both VR and AR as the spatialized web. That’s the idea that our internet connection and existing communication collaboration platform is going to expand to offer more three-dimensional content as well as multisensory, immersive, and ambient interfaces. What we’re creating is experiential media technology. We’re now using our bodies to communicate. We’ve done a tremendous amount of things using the 2D computing interfaces that we’re using right now to communicate. But the paradigm of working with windowed, two-dimensional, informational content is going to be shifting into computing experiences that are multi-dimensional. They can even add dimensions you can’t experience in real life. I’ve walked through six-dimensional space in virtual reality before.
My entire career has been looking at emerging media technologies and asking the questions: What is the conversation we could have now that we couldn’t have before? What can we say now that we couldn’t say before? And within the paradigm of VR and AR those include physical experiences and this idea of embodied presence or embodied cognition. That’s the understanding that we don’t just think with our minds. We think with our entire bodies, and we think with our environment, and we also think socially. These ways of thinking are not necessarily well supported by our previous tools of communication and collaboration.
What might a school or workday be like if everyone had VR or AR?
Before I say that, I want to say that I’m not suggesting that VR and AR will replace all previous media. I also don’t think they will replace in-person communication. That said, virtual reality is amazing because all learning is based on experience. Whether that be your own direct experience or experience communicated to you by others through some sort of media. What we’ve developed through history is representative, symbolic, or abstract. We’ve done so much with that. But really it’s been dictated by the limitations of the media. What VR and AR are bringing in is this idea of experiential media. What you’re going to be able to do is give actual direct experience rather than just describing things. Secondly, we’ll be able to have these collaborative virtual spaces, eliminating the barriers of the need to be in the same physical space.
There’s a famous quote by Clay Bavor who runs the VR/AR team at Google. He says, “VR can put you anywhere, and AR can bring anything to you.” Really what that says is not only can we go anywhere, we can go anywhere, at any time, at any scale — from a molecule to staring back at the entire universe. We can go there with anyone. And we can go as many times as we want.
So what are some interesting applications of that in the real world?
For training for a doctor, simulation is really powerful. Often they need to learn a real world procedure that may only happen once a year in a hospital but is a life threatening moment that they need to be prepared for. And they’re able to recreate those experiences.
On top of that, you can not only go anywhere, you can be any perspective in that experience. Going back to the doctor, imagine you’re trying to learn how to diagnose a patient who has come in with a gunshot wound in an emergency room. You can run that as many times as you want, but you can also run a recording of yourself doing that from the perspective of the assisting nurse, and you can see what you’re like in that experience. You can step into other roles. You can run it from a patient’s point of view, or from the point of view of a loved one in the waiting room. You can run it even from being inside the patient’s chest if you want.
VR is sometimes talked about as an empathy machine. I think that’s a little bit inaccurate. Like any media, it has the capability of inducing empathy, but this requires good story telling. It offers a sense of presence and perspective. You can have multiple perspectives, which I think is really valuable.
The other experience people may have is more about augmented reality. You could go to school and acutally go to this abandoned lot down street. You may think there’s nothing here, but we’re going to go learn all of the wild plants in this space and your role is to identify what all those plants are and mark those and turn that into a virtual reality experience you can share later. Or you may do a walking tour of your neighborhood. And you could learn about all the historical ages of that place.
So as you’re walking through the abandoned lot, you could see additional information identifying plants. Or as you’re walking down the street, you’d also see images or video about the past history of that place?
Exactly.
How might this interface work? How are we going to be accessing VR or AR in the future?
For augmented reality, the contact lens is something that many people have been dreaming of. That’s pretty far off. I’d put it at 10 years or further. Because there are a lot of technological challenges there.
As AR gets more sophisticated and higher fidelity, I think VR will become more of a niche interface. It’s very useful to be able to simulate virtual environments, there are a lot of applications. But doing everything you’re already doing with the ability to have additional information is exponentially more useful. I’ve had a chance to try some of these. What we’ll start to see are smart glasses that are sophisticated and hook up to your phone.
There are a lot of challenges in the hardware for these technologies. The more important challenge is what is going to be the content. It’s a chicken and egg issue. There’s a lack of investment in developing content because there aren’t a lot of users because the hardware is so limited. But people don’t want to invest in developing more hardware until there’s more content.
One thing that might be interesting to your readers is that we’re looking at a transition to content on these platforms being more user generated. Just as currently happens on the internet [with video, audio, images, and text]. Technical barriers for people to create AR or VR experiences are quickly dropping away. There are lots of non-code-based tools for creating that content.
So VR is going to be for things like gaming or professional training, whereas everyone is going to want the ability to see information around the things they do in their daily life. We all have our phones for texting and social media now. Instead, we might have AR for doing that type of communication while VR will be something you go into on occasion?
Exactly. It’s not a perfect analogy, but we don’t go to the movies all the time. People don’t want to be fully immersed all the time. That said, there will be glasses that integrate both. Oculus Quest has the capability to do both. There’s a term called XR which refers collectively to VR and AR. People interpret it as extended reality or extreme reality. I like “cross reality.” The fundamental concept of all these technologies is some form of data visualization combined with some form of spatial tracking. The real key is to make a meaningful connection between our digital world and our real world.
Let’s talk a little about the technical barriers. What is going on in technical development that is holding us back?
In VR, we’re limited right now around field of view and how wide that is. There are lots of problems with optics and seeing stereoscopically, but where our plane of focus is never changing because we’re looking at a screen. Disorientation comes with that. A lot of people experience simulation sickness when you have even a slight mismatch between the inner ear and what you’re seeing visually. Those are some of the existing challenges with VR from hardware standpoint.
On augmented reality, you have all those problems. Plus you want portability. And trying to synchronize your visualization with the real world is much harder to do than replacing all of reality. When it comes out of alignment, it breaks the sense of presence and realism. For AR, people want something that’s wearable. So the size and the battery is a really big problem.
What are some of the social changes that might take place as more people use VR and AR?
We did an ethnographic study of social VR and we looked at user behaviors. Like, what’s it like to wear virtual avatar? Or how do you manage, psychologically, the idea that you’re in multiple realities and have multiple selves that are embodied? When you’re embodied, there’s a very different sense of connection to those experiences. The same way we’ve had to develop our sophistication around other media formats, it’s going to take us a while to build not only the fluency to create it but also the literacy to understand it.
This is all going to merge with AI. Sometimes we’ll be in bodies controlled by us, or by models of us. We’ll be interacting with people who are sometimes themselves, sometimes fully automated characters, and sometimes hybrids. Some people will be able to control multiple characters simultaneously.
What are some of the ethical issues with VR and AR?
All of the things that are going to make VR and AR more immersive, give you more of a sense of presence, more contextual, more useful, or more personalized, unfortunately these also happen to be the exact same things you’d use to develop world’s most sophisticated surveillance system!
This is a huge conundrum we have. We’re entering into what some people call surveillance capitalism. It would be irresponsible not to say that there are a lot of problems we’re facing. This is about to really transform human consciousness and really rework the meaning of our economy. The economy of behavioral data is about to be as large a driving force as the industrial revolution, in my mind.
That’s right, the whole point is that the system is surveilling you and figuring out where you are and what you’re doing in order to show you things and show you experiences. That could be used for bad reasons as well as good ones.
It could, and unfortunately, it will.
One of the ideas that Ray Kurzweil promotes is the singularity, the idea that we’re going to merge with our machines. What do you think of that idea?
Personally, I think that we are already merging with our technology. We are cyborgs today. I don’t know my girlfriend’s phone number anymore. We depend on machines and we’ve expanded ourselves.
I think one of the things that’s a little hard to see outside of is the paradigm of the modern sense of self. Even the idea of intelligence as being individual. I think the internet is helping us to see ourselves more collectively. The idea that we’ll create virtual versions of ourselves, I think is already happening. We’re about to see a whole new class of virtual persons.
We are not just going to be merging as in a brain in a jar with electrodes in it, or a brain downloaded into a computer. We’re going to be modeling our personalities. This is maybe not merging, but interpreting. And in interpretation, much is lost and much is embellished.
My view may be not as utopian as the view that Ray has, that this is going to be beneficial for all. I think what we’re clearly seeing is increasing inequality. I think lots of very wealthy people will have amazing technology to allow them to live very long and give them the sense of living on forever in electronic form.
Many of the technologies that the Institute for the Future and Ray have been talking about for decades, like artificial intelligence, like virtual reality, like ubiquitous computing, those have all been down the line, and now suddenly they’re here. They are leaving the labs, they are entering into society, primarily as commercial products that are being deployed at massive scale before we really understand what they’re doing. So I think we’re going to merge, but I think we’re merging in ways that aren’t entirely intentional.
Any last words?
Just to reiterate, developing literacy in this space is going to be more crucial than ever. One of the things about this technology is it’s even more persuasive and potentially dangerous [than current media]. You can put down a book, and it’s never going to do anything. These technologies are following us, and learning about us, and actively grabbing information about us to try to persuade us. The reason there’s a market for it is to persuade us. Young people are surprising us all the time. I’m hopeful that they’re growing up as XR natives and will be more savvy around this. But one of the things we’re learning is just how hackable humans are.
In December 2020, I caught up with Hoo again. We talked about how the pandemic had altered the landscape of VR and AR.
We’ve called it the global communications hackathon. Everyone had to figure out a way to get connected, collaborate, and connect using available tools. People are lacking and missing the embodied experience of being in rooms. You realize a lot of things when you suddenly don’t have them. There’s been renewed interest in social VR.
There’s a lot of hope and optimism that we will at some point be able to gather in person again and that will change things. But for many reasons we’re not going back [to exactly how it was before]. All aspects of life have been forced to adopt digital formats. What we will likely see even when we go back is a much more hybrid model.
On the negative side, a massive amount of human data that was untrackable is now translated into structured data. A new raw material is becoming available which is human behavior.
One thing I’ve missed in having only virtual interactions with my family is hugs. Is that something we could do in VR? Could that feel real?
We’re working on a project right now with the Alameda County Probation Department to prototype and pilot use cases in VR for people being released from prison. We are doing some filming in VR. One of the test shoots I did, it’s a format called 180 VR. It’s not 360 degrees all the way around, but we’re able to do it in stereo so it’s 3D. I did some filming tests with my fiancée. I said, “Hi, how are you doing?” then we went forward and hugged. Then I went through the process of putting it into VR. It’s pretty powerful. You obviously don’t feel the physical sensation, but especially when it’s stereoscopic and photo realistic like that, as you approach, you feel the chemical releases of a hug.
Obviously, hugs are age-old for human beings. [But there are many other ways to feel a connection with others]. This year, we hosted one of our annual conference events in social VR in Altspace. We hired facilitators who proposed doing these icebreaker games. To be honest when they first proposed it, I thought it was a little bit cheesy, but I took their advice. They did it and it was actually very powerful.
In AltSpace VR, you have an avatar, and you have emojis or emoticons. You can clap, you can heart, and they pop up above your head. [Most attendees] weren’t very comfortable operating emotions through that awkward interface. So the facilitators played a game. They had the whole group together, and one person would come up on stage and share one thing they love. I’d say, “Hi, I’m Toshi, I love dogs.” And then they’d say, “Who else loves dogs?” [Then other people would use their emojis to agree].
It sounds simple, but I call it emotional proprioception. They were trying to do the mapping for when I feel affinity for somebody or hear something I love, I’m going to do this behavior. As you bring experienced designers and facilitators of human connection and bonding into these spaces, it changes how they’re used. You can build a word processor, but it will never quite fulfill what it is until you bring poets and novelists in there.
This year, Burning Man was held partially in Altspace. What that meant was over 150 art worlds were created in Burning Man with the same sort of creativity you might see at real Burning Man. It’s a very different medium, but there are now some amazing things in there.
I think that’s a really important point. Maybe you can’t give someone a real hug in VR, but there’s things you can do that you can’t do in real life, like project some emotion outward from your head or change colors or something.
Totally. As a case in point, there’s a social VR platform called Wave which is a concert and performance platform. The concerts are very cool and innovative, but they have this area outside of the concert venues. It’s like a tailgating area. They have these virtual toys there. One is like a light stick.
If I’m using [the light stick] by myself, it acts one way. If someone near me is using the same toy, they both go up to another level that you can’t get unless someone else is doing it. And you get a visualization of this kind of lightning-energy connection between your chest and their chest. If a third person does that, it goes up to a higher level and you get this triangle. Of course everybody else can see this, and they’re like, I want to be connected. It’s not a hug, but it’s a pro-social visualization to try to encourage people to want to engage with each other and find a connection.
That is so cool! Another thing I was thinking about is the idea of risk. On a real hike for example, you could run out of water or fall off a cliff. What does it take away from a virtual experience if you know that you’re going to be safe?
Obviously there’s a danger of escapism and a danger of losing a sense of consequences. But are you looking to replace reality, or supplement reality? Is this a practice space, or a place for me to hide? Is this a place for me to connect with other people and discover new parts of myself, or is this a place for me to go in and harass people because I have the anonymity to hide myself? It’s a lot of the same issues of the internet in general. But it is multiplied by the fact that it’s an embodied experience.
As a replacement for reality, it’s terrible. But as a supplement to reality, as a practice space, as a connection space, it’s excellent.
We’re not trying to build the Matrix and replace reality, but to increase our options for how we connect and share experiences. That’s very cool!
Exactly.