It’s a while since I wrote my last story…and that’s because I am really… really… busy. I am currently working on my master's degree thesis, which consists in creating a 3D web-based virtual experience.

Last month, I did a lot of research on what’s the state of art of virtual experiences and multimedia technology, finding out a lot of interesting things. Did you know that in Japan you can visit an exhibition so technologically advanced that you can draw creatures on paper, which are then brought to “virtual life” with technology?

Let’s see how beautiful technology is today, and how you can design the ultimate experience.

A forest where gods live.

Inspired by nature and how technology endangers the environment, this project by teamLab is an incredible piece of art. By visiting the Mifuneyama Rakuen Park, created in 1845, now you can find out how Volvo cares about being “climate neutral”: dozens of installations merge the power of light, sound, and art with nature as its canvas, creating a unique environment which empowers the transition between anthropic environments and forests.

But the most fascinating part of this exhibition is interactivity. There is a special place inside this park, which allows users to collaborate and interact with the installation: Graffiti Nature.

Graffiti Nature is an installation composed of numerous interactive surfaces where visitors can walk and interact with a living digital ecosystem: by stomping on lizards or staying still, you can decide on how the ecosystem (that also works by itself), evolves.

But as I said before, there is more. You can draw these magnificent creatures.

This is cool but why is this characteristic so important? Ever heard of the Cone of Experience?

The Cone of Experience

In the ’40s, Edgar Dale defined his theory about the Cone of Experience; this theory basically tells that the less you abstract an experience, the more the user will remember and interiorize what he experienced.

Often, you’ll find images like that one up here, but beware: the percentages are false. There is a scientific demonstration that this theory is correct, but there isn’t a well-defined line between the different levels of the pyramid.

Let me correct that image:

The drawing part of “Graffiti Nature” is extremely important because it lands on the base of the Cone of Experience:

doing teaches more than everything else.

Remember this for later.

Web experiences

As this story’s title suggests, we should find out how to create meaningful WEB experiences. Obviously, Teamlab’s case study isn’t a good example of this kind.

When designing a virtual experience, is a good idea to find out what already exists, so you know:

1. what you can do
2. what can be improved
3. the limitation of a determined technology

Since I am a designer but also a coder, I decided to design and develop my 3D virtual experience using a language I already know: Javascript.

There are various libraries and platforms that allow you to design 3D experiences, like Unity, P5.js, BabylonJS, and ThreeJS. Personally, I’ve played with Processing many years, and let me tell ya, It’s really bad for these kinds of things.

Unity is a great option nowadays, but you have to deal with 3 major drawbacks:

1. You have to know the engine (a very specific kind of coding)
2. You maybe have to connect the website DOM to it (easier in JS)
3. Could be pretty heavy, download-speaking

Since I know nothing about Unity development, I preferred to stick with ThreeJS…so it was time to find out the state of art of ThreeJS 3D experiences.

Well..there are a lot. Here are my favorites, and what both me and you can learn from them.

Life in Vogue

This case study is a great example of how limited could be a web-based 3D Engine. Based on WebGL, this 3D experience had to deal with rendering, and how hard is to get notable results with “minimum” effort.

Monogrid, a great Italian web agency, used a lot of tricks to achieve photorealism, good performance, and web. In fact, The first part of this experience isn’t really 3D, but a pre-rendered video that is animated by user input. Creating such a complex scene in WebGL would require A LOT of effort, especially because it’s not a ray-tracing engine. Basically mission impossible.

By continuing the experience, we can visit different rooms, that also use a trick to achieve high-quality render: cube-mapping.

For example, this room up here isn’t actually present inside WebGL, but it’s a simple cube with a particular texture. Monogrid rendered this scene in Unreal Engine, exported it as a 360 image, and placed it as a texture inside webGL. Minimum effort, great results.

Downsides? since it’s fake, you can’t navigate it.

We’ve found out the first problem with web-based rendering: you can’t achieve great live renderings.

Hollow

A completely different example is Hollow, a 3D virtual artwork by Katie Paterson.

In this virtual experience you can actually navigate a 3D space, but both for artistic and efficiency reasons, the scene isn’t really complex.

In the first scene, you can interact with only a couple of objects: ThreeJS doesn’t have an optimized raycasting system so dealing with thousands of items could dramatically slow down your app.

Both two parts of Hollow’s 3D experience are navigable, and developed both to easily work on desktop and mobile.

One of the major issues of a 3D experience is in fact creating a pleasant navigation system that works both well on mobile phones and computers. By limiting the degrees of freedom, Hollow is a nice example of how you can achieve high both memorability and accessibility.

Let’s do a recap:

1. Interactivity is a great way to make an experience memorable
2. Web-based experiences lack high-quality rendering techniques (nowadays)
3. there are many tricks to fake a 3d environment like cube mapping
4. keep it simple, if you don’t want to show off. Remember that the environment should work well both on mobile and desktop…and the first one is hard.

My take on 3D virtual experiences: the Infinite Museum

After this long research (I just cited a couple of examples…), I found out what the limitations of my choices are…and what I should do to achieve the most memorable result.

My idea is to create an infinite museum, which self-generates based on random code. It currently works, but there are too many flying artworks around…I’ll find how to fix the bugs.

1) Interactivity

Since interactivity is a must, I decided to create two different ways to make the visitor feel part of the project:

1. The visitor’s visits and paths change both the museum and the live artwork shown on the website
2. The visitor participates in the museum's look and aesthetics by adding items (trees, flowers…) to it.

The first part already works, but the second will require a lot more coding…

Obviously, the visitor can click and find out information about the exhibited artworks!

2) Rendering limitations

The limitations on 3D rendering are forcing me to look for a minimal aesthetic, and I have currently decided on a surreal and dreamy environment where the user is both lost and calm.

Since the concept of flow, infinity, and evolution… I think that this surreal atmosphere is a way to both empower the message of my project, and fix the technology problems.

Speaking of code, I am not that great but I’ve implemented a couple of optimization techniques that allow me to get over 10000 ( basically over a square chilometre) rooms with 200FPS at ease. I think it’s enough, ay?

3) Tricks to fake 3D

I want to show off. No tricks or gimmicks. My entire world is real. The only “trick” I use is to simulate raytracing with a particular map technique…but we don’t care about deep coding tricks.

Also, I want to make everything fulling navigable… so ->

4) Accessibility

This is a tough part. Coding navigation systems, if you hate trigonometry like me, it’s pretty hard. To avoid this kind of problem, I mimick the most pleasant kind of navigation patterns I know, like the FPS games ones and mobile games.

Remember always that people are used to particular patterns, and using them is a good idea to increase accessibility. This both works for classic websites, apps, and also games and virtual experiences. Yeah, people do not visit 3d museums often, but play a lot: copying videogame patterns is a good idea.

Currently, both mobile and computer navigation work and the project is navigable… but a lot more can be done.

Some real and precise tips to create a virtual experience

Let’s see again a recap of things you can do to achieve great results by designing a virtual 3D experience.

Design remembering interactivity.

Always think of a way to make the user interact with your masterpiece. You maybe have a nice idea, but could be simply a show-off with nothing memorable.
By making the user participate, you can create an extremely memorable experience, and probably he/she will tell others about how he/she contributed to creating an artwork.

Find a way to make the user participate in your work. It’s not that hard. Even simple things like moving can be used to generate random numbers that later come in handy. Remember to let him know that!

Design remembering limitations.

Check up others’ work. Probably if you want a photorealistic experience in Javascript, your developer (or yourself) will get angry. We are used to fascinating graphics nowadays, and working with web 3D engines could be disappointing.

Look for minimal or cartoony aesthetics, especially things that do not require complex lighting. (Lighting is in fact extremely slow to compute…beware).

Design remembering accessibility

Creating an accessible experience is what could lead you to most withdrawals. Raycasting and mobile navigation could be your worst enemy.

Sometimes, if real 3d walking-around isn’t that important, go for simple click-and-drag kinds of things. Look at this example: great aesthetics, minimal navigation.

ThreeJS includes orbit controls which could be enough for most experiences. If you want more, prepare to pay or sweat more.

Also, if you want to include VR (Virtual Reality), beware of accelerations. You should never include acceleration inside a VR experience because it leads to nausea very quickly. This could really change the way you design navigation patterns.

Test your damn 3D experience

Test it, test it, test it. 3D experiences are a lot more complex and could hide many glitches. Passing through walls, exiting the map you intended, etc. Test it and ship it only when you’re extremely comfortable with it.

Also, make someone else test it. Maybe you find some controls intuitive, but they’re not. Study how video games deal with controls since they’re the most common software people use for entertainment.

Now, have fun and try creating something. Start with something simple. Once you’ve grasped the basics of an engine like ThreeJS… you can create even minigames or original portfolio websites!