Last year, I decided to start an article series to share my journey of learning Computer Graphics(CG). I have shared some thoughts on why you should learn CG as a VRAR Practitioner. In this one, I would like to tell my story of why and how I started learning Computer Graphics for creating good VRAR content. In the end, I will show you my 5 learning strategies and share some amazing learning materials with you to get you started.

Why did I start learning CG for creating VRAR Apps?

It was 2014 when I first tried on an HTC VIVE VR headset and left my first VR painting using Tilt Brush in what we call now — the Metaverse. 😍. I fell in love with this unknown field at first sight. At that time, I was studying product and interaction design at college, when most of my projects were about 2D UI and UX design. VRAR opened a new world to me where I, as an interaction designer, could create a whole 3D world instead of several flat UIs on a small mobile phone. I began to seek any chances I met to play around with VRAR, dreaming about being one of the pioneers to create a new design paradigm for the next generation of computing.

As I’ve been involved in this industry, I started to see the difficulty and complexity of designing a solid virtual world more and more. In the beginning, I was pretty confident about being a VRAR designer because I have a good track record of building traditional 2d apps. I also knew how to do modeling and rendering with 3d authoring tools, and I learned a little bit of game development in Unity as well. I thought those skills should prepare me to create good VRAR content, but the reality hit me hard 😵.

Back in 2015, when I created my first VR app, I met my first technical blocker, and in the end, I had to change most of my design. I designed a fancy house with glass walls on all sides to welcome the new users onboard in my VR app. It looked perfect in Unity — the engine I was using for VR development, but when I was running it with my VR headset, the view of the scene was jittered. Especially when I was moving my head fast, the image even became freeze. Later, I found out that the glass material I attached to those walls was the culprit. Somehow, the app was not powerful enough to support this material rendering in real-time while maintaining a high frame rate required by running a comfortable VR game. And I had no idea why the frame got impacted by this material. To avoid the complete failure of delivering this project, I replaced the whole house model with another one. I started to wonder why changing materials such an easy thing in a 2d app becomes so computing costly in VR?

And not just material choice. Even color adjustment has become tricky for me when it comes to VRAR. In 2d app design, color changing could be just a single click from a color wheel in Photoshop. But in the world of VRAR, Value representing one specific color is just one single variable among ten others in the pipeline of generating a colorful 3D object. In fact, the texture attached to the materials using one particular shader, the rendering pipeline, the GPU, the screen, and your eyes would all contribute to the final color seen by you, which I didn’t know at that time.

https://learn.unity.com/tutorial/shader-graph-vertex-displacement

My blind spots were the technical constraints embedded in the gene of VRAR technology, and my history in designing 2d apps blinded me from them. Frame rate per second(fps), for example, represents how quickly the monitor can update all the pixels’ colors on display to draw one image. It is measured in Hertz (Hz), which means if the value is 100Hz, the monitor can refresh the image 100 times per second. 24 fps is the bottom line for human eyes to see moving pictures. If the frame rate is lower than that, we will see discrete images instead of motions.

Frame rate is not a thing that we usually worry about when designing a mobile APP or website because nowadays, computers are powerful enough to support most flat screen interactions. However, when it comes to VRAR, the bottom line raises to 60 to trick our eyes into 3D. And 90 fps is desired number for a smooth 3D experience with low risks of motion sickness.

The difference in frame https://artlist.io/blog/how-to-use-frame-rates/

The 90 Hz Frame Rate is a huge constraint that needs to be handled very carefully, which is supposed to be common knowledge for a VRAR designer. I, instead, didn’t get lucky to know this when I started my immersive journey several years ago. I’ve spent years and been through many emotional breakdowns and design compromises to figure out what that means behind the number and how to make a wise performance and quality tradeoff for VRAR applications.

To me, at that time, there were so many uncertainties in the process of creating VRAR content. So I listed them down before I could tackle them:

What variables will impact the frame rate?

How does each frame get painted in colors?

And How to achieve a photorealistic effect for a 3D scene while maintaining a good frame rate?

How does a VR headset trick the user into thinking the virtual world is real? How does an AR headset trick the users into thinking the 3D object is located in the real world?

While searching for answers to those questions, I found that all information pointed me to one interesting subject — Computer Graphics, and I didn’t know much about it. I knew that was the gap I needed to fill, and I started my study without hesitation.

How did I learn CG for VRAR?

Computer Graphics (CG) is a complex subject to learn by yourself. My self-learning journey is a bumpy experience. So might be yours. But I assure you the time and efforts you spent to learn in your own way will be worth it. I started from feeling that the graphic pipeline for creating a VRAR product is full of unknowns and uncertainties. After three years of learning and practicing CG, I’m getting more capable of tackling most of the performance and quality tradeoff issues that happen to my daily VRAR works. I can finally give a satisfying answer to the question — what variables can impact the final frame rate on your immersive headset.

When I reflect on my learning journey and think about what got me here, I identify five steps I’ve followed to tackle this complex subject. In the following section, I’d like to share these five steps with whoever wants to start learning CG now, either for their VRAR projects or simply interested in CG. And in the end, I hope my learning strategies can shed some light on yours and draft a concrete learning plan for yourself.

Step 1: Set the Learning Objectives

Learning objectives are the anchors that keep you from getting lost in a long learning journey. Beginning with the end in mind, as suggested by Stephen R. Covey in the book The 7 Habits of Highly Effective People, tells us to picture what you want to achieve in the end for embarking any long-term enterprise or project. I applied this suggestion to my case by having a list of questions I wanted to answer and problems I tried to resolve at the beginning of my learning journey. You might feel they’re valuable, so I share them here:

In the context of VRAR,

1 Performance. I want to understand the technical constraints. In other words, what will impact the performance of a VRAR application? More specifically, what may affect the frame rate in every sense of the pipeline. What should I do to meet the 90FPS(Frame per second, the indicator of how well your app can perform)?

2 Quality. I want to achieve pixel accuracy for my VRAR project; I can achieve the same effect I desire in my mind. So the question is:

How does each pixel on the VRAR headset’s displays get colored during the runtime? What is the mechanism behind a photorealistic virtual world?

3 Tradeoff. And more importantly, I want to understand the tradeoff between performance and quality.

How to achieve a photorealistic effect for a 3D scene while maintaining an ideal frame rate? I want to know what alternatives do I have.

See if you can answer these questions; if not, you might need more knowledge to fill the gap.

Step 2: Identify the Gap

Where is the gap I need to fill, and What is it? To figure it out, I pulled a blank paper, wrote down the questions I wanted to answer, and tried my best to answer those questions with the knowledge I had. And then what I didn’t know seemed nowhere to hide.

It’s a quick self-test that helped me evaluate my current knowledge level. I used it when I started my journey and have used it again and again since then. Our time and energy are limited so we really should decide which things we want to spend more time on. I want to pay more attention to areas I don’t know. They’ve changed as I learn more, and then my focus has shifted with them as well.

Step 3: Build the Scaffold

To be honest, identifying the gap was harder than I thought initially. Since, commonly, we don’t know what we don’t know. It’s our blind spot. We need some strategies to find our blind spots.

My way to locate blind spots is to build a scaffold that shows me the connections between knowledge. The learning process is like collecting dots, and only when we link them together, a collection of dots becomes the knowledge we own. The order of how we get there is critical. We can only comprehend a particular topic with the premise of knowing some other knowledge already.

So how to decide the scaffold — the learning route I should follow to get the dots I want to have? My approach is to take many beginner courses online like CG 101 from famous CG people or colleges that can give me a concise but comprehensive overview of the key concepts and theories in CG. Luckily, many good courses are out there. A good beginner course can make you do more with less. Here is a list of my collections:

CG Study Materials 1

The following are the scaffold I built for my early stage of learning:

By following this scaffold, I’ve traced down to the unknown space of my CG knowledge. For instance, Pipeline:

• I have a good knowledge of humans perceiving colors but I don’t how CPU and GPU work.
• I have knowledge of modeling but I don’t know the industrial standards and the behind story in terms of how to create models mathematically.

After I located many of my weak spots, I then tackled them one by one along the way.

Step 4: Pick the Accessible Point

A wise way to climb the learning scaffold is starting from our accessible points — the topics we‘re familiar with, and then using them as the footstones where we can step on to reach the next unknown point.

In my case, my entry points to CG are Graphics, HCI, and VRAR, which are three areas I have good understandings of.

👩‍🎨‍ Graphics is one thing I’ve been interested in since I was a kid. I have gone to weekend art school to learn sketching and painting for many years. The drawing skills I gained turned out to be very useful to understand many drawing algorithms in Computer Graphics. Things like sketching a smooth curve and drawing with perspective are pretty similar in the world of human painting and the world of computer drawing. I thought learning CG needs good Math skills initially, but when I approached this topic from the perspective of Graphics, it now became a field that studies how to train computers to learn how to draw and what to draw. I realized that the key questions I want to answer could be transformed into the following twos:

🖥🖱⌨️ CG also has its roots in HCI (Human-Computer Interaction).

[1]DEC 340 display

At the early age of HCI study, researchers created displays with graphic interfaces for engineers to work on the computer[1]. The technology behind it contributed to the birth of the first video game[2].

[2]The first video game — a tennis game

I studied HCI in college, which helped me understand the interactive part of CG and the human visual and cognition process. It shows me an overview of the techniques to capture human motions and return feedback across the different human sensory channels. In the context of VRAR, it provides some background knowledge and potential solutions for constructing a realistic virtual world where we can make you:

• 👁 See 3D.
• 🧟 Have virtual body ownership.
• 🚶 Feel present and free to move around (locomotion).
• 🤹‍♂️ Have controls over object selection and manipulations.

If you feel that CG is a tedious or complex subject to learn, you may start from the wrong place. Learning is a process that we’re gradually building the connection between the new and the old. If we find something difficult to learn, it is because we fail to develop connections. You can approach CG from diverse angles based on your expertise and objectives.

• Drawing can help you quickly grasp how the computer shades a mesh.
• Optics Physics — how photons bounce off different types of surfaces to create this colorful world — can help you comprehend diverse reflection models and rendering methods.
• Human Visual Process — Understanding how human eyes see objects can connect you to the underlying reasons why some hardware or algorithms are designed in that way. For example, why colors of pixels are represented by RGB(Red Green Blue) and why the virtual camera of a VR app should set the field of view angles equal to 60.
• Linear Algebra and Vector Math can show you the calculation help you understand a lot of transforms in the pipeline.
• If you’ve been a product designer, game developer, or film VFX maker, you’ve already been in the express lane. All you experience from using 3D modeling, rendering tools, or game engines can definitely guide you easily.

To step into the unknown, we’d better start from some places we knew well.

Step 5: Establish the Study Routine

To apply what we learned to our works, we need to reinforce what we learned again and again through a solid routine.

Here is the routine I followed:

① Learn and Reflect from time to time

• Create mind maps that capture key concepts and theories for each topic like render, model, animation [4].

CG pipeline Mind map

• Write reflections on what I learned from courses and projects.
• Test my current understanding by answering the initial questions I raised.

② Learn from the great

• Collect use cases + their solution plus analysis.
• And here, I’m aimed at answering my third learning question: how to tradeoff performance and quality? I want to know what options I have, and I want some alternatives.

③ Learn by doing. Practice.

• Have the toy projects that I can apply what I learned. I like creating VR landscapes or hand drawing apps using the CG techniques I learned.

My VR Landscape Showcase

Learning by doing. Theory plus practice. Too many theory explanations will push you away from the true beauty of this subject and how to apply it to the real world. Too practically, it will fall into the problem of superficial understanding, which is devastating for VRAR projects, cause the technology is not mature enough, More challenging problems require deeper understandings of the basic concepts, being a VRAR game engine master may not be enough.

Hey, that’s all! Thanks for taking your time to read! Here is a template I used to create my CG learning plan, feel free to grab it and make it your own.

CG Study Plan

Enjoy your learning!