Jaron Lanier invented Virtual Reality and coined the term at VPL in 1987. Virtual worlds require high-resolution 3D graphics and low latency to feel immersed, present and co-present with others.

Those ideas are widely believed and oft repeated, but are mostly bunk. Let’s take them apart, unpack and hopefully debunk.

“XR” is just our shorthand for all variations of digital realities: AR/VR/MR and beyond. The “X” means “any,” though some people also say “eXtended.”

Based on the best on-line research money can buy, the term “virtual reality” was actually coined (in French) by Antonin Artaud in The Theatre and Its Double (1938), described pretty well in “The Veldt” by Ray Bradbury (1951) and later invoked in The Judas Mandala, by Damien Broderick (1982).

Of these three, I admit I’ve only read “The Veldt.” I love Bradbury. But I’d also recommend “As We May Think” by Vannevar Bush (1945) and True Names (1985) by Vernor Vinge to understand the history and evolution of the ideas behind modern XR. Dr. Vinge, retired professor of Computer Science at UCSD gets right into the mismatch between worlds of almost literal 3D magic vs. the realities of loneliness and disempowerment IRL.

Snowcrash (1992) by Neal Stephenson and Ready Player One (2011) by Ernest Cline are also very fun books, but not really guiding architectural XR documents, no matter what some wealthy fans say. The books are extremely dystopian, which should be a giant warning sign for anyone trying to build them. Even in fiction, these worlds had big sociological downsides.

Revealing your real name in VR was very dangerous, because you are so vulnerable IRL (you could respawn in VR). Globally powerful forces like IOI or a thought-virus could more easily attempt to own the virtual universe than the real one, enslaving pretty much everyone. Fun stuff!

Ironically, the authors seem eager to build their own fantasies IRL. My first startup met with Stephenson in 1994 to consider the feasibility, too early. He worked for Magic Leap as a futurist from 2014 to 2020. There are early incarnations of “The Oasis” for VR headsets now. I can’t complain too much. I’ve worked from 1992 to now trying to make something I hope is better for humanity, and maybe myself.

So where did it all really start?

The most important early XR work IMO was Ivan Sutherland’s 1965 paper and 1968 demo called “The Ultimate Display.” It’s often misremembered as the “Sword of Damocles” — that was the name of the large overhead mechanical tracker, as seen in most photos, which was used as a fallback to his wireless tracker, which was still too janky.

Sutherland realized, in 1965, that this approach could ultimately be the one personal display to replace all monitors. It was a working AR demo that proved the concept of head-tracking coupled to real-time 3D rendering to create a personal 3D view on top of reality. And it’s taken 53+ years to make it to a consumer product. Turns out, it’s not so easy.

For more pure VR approaches, the “Sensorama” system from Mort Heilig came even earlier (1962) but was even less well known or successful. It’s often depicted as a large stationary VR device you’d sit near and lean into, for a so-called “4D” experience. His wife has shown some evidence of an actual head mounted display as well.

Jaron Lanier contributed many creative and original ideas over the years and deserves ample credit for his work. But there were also other less well-known pioneers at VPL, NASA and elsewhere at that time. Jaron has been interviewed extensively and written many thoughtful books around the intersection of technology and society. I got to work with him at Microsoft, a bit before I helped start HoloLens in another part of the company.

Virtual Worlds

“Virtual worlds” is the concept that underlies every janky XR term you might encounter. The ideas have been studied for decades, both academically and in the industry. If you’re looking for the one term to rule them all, this is it.

But virtual worlds are not what you might think.

Ceci n’est pas un chapeau, by Magritte

Virtual worlds aren’t the headsets, controllers, or other devices we might use. Those are all physical interfaces.

The same even goes for what’s being rendered via some head mounted display. Those are virtual interfaces — and not just the GUIs we see. Literally all of the 3D sights, sounds, haptics and whatever we experience in XR are not the thing itself, just a digital copy of the thing, like cue cards, meant to invoke the ideaof the thing in our minds.

Virtual Worlds exist primarily in our minds. That’s the key idea.

They exist in everyday conversation between two or more people IRL.

They exist when we’re alone, just thinking about the world.

Virtual worlds are a part of consciousness itself. They are our mental models for the world, including any imagined concepts or ideas. We never experience reality directly, but only through our sensory organs and our memories, learning, interpreting and reinforcing these virtual worlds of thought.

This is why synthesizing 3D audio/visual content works so well. Our minds construct equivalent virtual worlds internally to reflect what we experience.

In communication, we translate concepts from one mind to another

Whenever we communicate, we translate the worlds in our minds into words, maybe sounds or text, which are later decoded by other people and turned back into ideas inside their own minds.

There’s always something lost in translation. When I say “car” you may imagine a totally different model or color.

That’s normal, especially if we don’t share context. But if we’re both standing by the same car, you might infer that I mean “this car.” The concept of “trees” may vary even more widely, depending on where we each grew up or live.

What’s even more interesting is that I can write “car” or “tree” on a computer and those few bytes can unpack the equivalent of gigabytes of detail in your mind. Language is the best (lossy) compression scheme ever invented.

The largely untapped power of XR is that it has the potential to translate our thoughts from one mind to another via a much higher bandwidth conduit than any existing language. But for it to really be about sharing our inner worlds, it requires even better tools for us to express our thoughts and enhance the feedback loop.

[Right now, folks are still working hard to convey body language correctly.]

So what is this “feedback loop?”

Step 1: Someone imagines something.

Step 2: They express it via some kind of tools, e.g., Blender, Unity, Maya, or ideally some direct interaction with tools in XR.

Step 3: A computer manipulates a shallow copy of these ideas that reflects or mirrors the original.

[In some sense, the computer or network has “virtual worlds” too. But they’re currently just data: numbers, equations, very little semantics…]

Step 4. The world at first imagined is later rendered in formats that we and others can perceive.

If the rendering matches our mental model, there is a form of consonance (think: harmony) or, if not, dissonance. Multiple cycles of this feedback loop can create some level of resonance (think: sympathetic vibration), and foster a sense of presence, and even co-presence when others come to join us.

What we perceive from the loop can also change the models in our minds. Did you ever see an object in low light and think it’s one thing, but then realize it’s something else? That’s the same feedback cycle in real life. The better we listen and observe, the more we understand the world and each other. It often takes active listening to overcome our initial mis-comprehensions.

Communication is mainly about annealing the differences in mental models among people (now with devices mediating). If you add more people to the mix, it gets even more interesting.

Pop goes the Ouija

In real life, a small group of people can move the planchette on a ouija board to spell words that none of them preconceived. Interestingly, it doesn’t work at all if you blindfold the participants.

This disproves the spiritual interpretation (unless spirits are withholding during all scientific tests). The surprising words we spell on the board are really coming from our unconscious, collectively transmitted through the slightest touch and some shared visual perception among the participants.

You might call it a small collective unconscious — an idea which potentially scales up to Jung’s for society overall.

Mental Models

Most of us can model the state of mind of other people, to varying degrees. I’m worse at predicting the thing I’m about to say will hurt someone’s feelings. But we all have these so-called “mirror neurons” to help.

Sociopaths are better at manipulating and saying the perfect thing to get their way. But they don’t see other humans as people. Rather, people are objects to manipulate to serve their needs. Non-sociopaths can form empathy, even for inanimate objects — toys for example, or 3D characters in movies. It doesn’t require immersion to have empathy or to believe in a personified object’s agency, but it may help.

Co-presence is the technical term for what happens when we begin to believe that the embodied agents we find in our virtual worlds are indeed real people (and not mere objects) who occupy the same spaces as we do. Most 3D collaboration services barely tap the surface of this mental feature.

When done right, people in different rooms or different countries can enter separately, spend as little as 5 minutes together virtually, and then exit VR, legitimately feeling like: “where did everyone go?” They are consciously aware they were never in the same room, but their mental virtual worlds included other people as present and fully real, regardless of visual fidelity.

So does XR require high FPS 3D graphics?

Imagine any great novel. You can be totally immersed in it for days with no art except the cover. So 3D graphics isn’t strictly required for immersion. Cartoon VR worlds also work, as do purely textual worlds, like text adventure games.

The main reason we have such high performance requirements for our XR interfaces is that doing anything wrong may distract us: the wrong gestures, heads, eyes, elbows; skippy frame-rate, wrong collisions (and more) can all break the illusion — the so-called suspension of disbelief.

Some of these errors can cause other people to seem creepy, not trustworthy, or even inhuman to us, which is worse.

Most notably, nauseating people with certain kinds of 3D graphics, where the visual cues don’t match our other organs (esp. inner ear), likely causes our bodies to believe we’ve been poisoned, for which its only known solution is to empty the contents of our stomachs on the floor.

Someday, we’ll have direct neural connections via computers that don’t require 3D graphics to be rendered at all. The goal, assuming the companies tapping into our brains are trustworthy (ahem), is direct transfer of ideas, from my virtual worlds to yours and back, without first rendering them into text or some pseudo-physical 3D form.

Even at that point, the goal won’t be for us to disappear into some computer-mediated world, but to bring our own mental virtual worlds into better harmony with one another — more resonance and mutual understanding than we can achieve through traditional communication alone.

Will it bring us together or drive us apart? I’ll explore that in a future article.

The UX Collective donates US$1 for each article we publish. This story contributed to World-Class Designer School: a college-level, tuition-free design school focused on preparing young and talented African designers for the local and international digital product market. Build the design community you believe in.