“Can you come tell us what user research does?”
Or…
“Hey, please let me come and tell you what user research does!”

We’ve all been there.

In other words, if you’re working in the industry today, I’m sure a big part of your job is evangelizing the role of user research. And you’ve spent countless hours communicating with partners on how you can work with them to make great experiences.

I’ve had the privilege of being in these conversations for over 15 years. A big chunk of that in gaming, where I started way back in 2001, and where I’ve landed now at Microsoft. I’ve even presented on the importance of packaging the way you work to product teams.

As you can imagine, in those 15 years our discipline of user research has changed. Shipping software and services, especially video games, has changed as well.

Big data is here and it is amazing. Flighting and beta programs are ubiquitous. The amount of data pouring in is staggering. And the number of researchers of all backgrounds at any given company is growing every year.

That means as user researchers, we have a much bigger sandbox to play in. It means we have the opportunity to impact the product more deeply and earlier than ever before. And it means we have to think about how we message user research in the new world.

So now when I talk about the testing phases of user research, I anchor it in “confidence.”

As we’re working on games, or any other product, there are a few phases. These have had many names over the years (and still do). The amazing design team I work with uses one of my favorite models: Creative, Iterative, and Reactive.

Essentially, the Creative Phase is where any idea is a good one. It’s the time to get crazy and think of anything and everything. The Iterative Phase has more focus. It’s focused on comps, specs, and general directions. But there’s still time to make big changes in the experience. And Reactive is that last mile before ship, plus everything that happens after ship — when the product is done and it’s time to tweak and improve based on feedback.

So how does user research fit in? How do we think about testing a product through those phases? And how do we message that to our partners?

I believe we can start from the perspective of our partners. Everyone on the product team wants to build something amazing that users will love. They look to research to help increase their confidence so the team can hit the design intent and the wonderful, innovative ideas we all start with translate into UX, then into customer love.

So I’ve created the graph below to help map out a view of testing products. It focuses on the Iterative and Reactive Phases since that’s usually where we’re doing the actual testing. (There’s tons of amazing research we can do in the Creative Phase. It’s more generative and strategic in nature, which is fantastic, but not the focus of this particular model.)

The goal here is to frame a few of the important factors in how we work with teams. Instead of listing them out, I’ve included the slide build and my talk track below.

Slide build and talk track

1. In any kind of game or feature, you can think of the Creative, Iterative, and Reactive Phases. For now, we’ll just focus on the last two.

2. The goal of user testing and user research is really to increase the confidence we have over time—to help the product team iterate on UX to help land the design intent. The farther out we are from ship, the less confidence we have. But with research, we can increase that confidence well before we ship. And even after we ship, we can increase our confidence that we understand the actual user experience in the wild.

3. To drive confidence over time, we often use three types of methods. We can observe potential behavior early on, then move to measuring attitudes and opinions. And then later we can measure actual user behavior and opinion once the product is flighting or shipped.

4. Since we’re researchers, we often think in terms of sample size. Each type of method has a certain number of participants. As you can see, the number starts small and builds, along with the confidence.

5. So all of this is important, but what’s really interesting about this model is where we can have the most impact on the UX. We all know late changes are more expensive. So while flighting and A/B testing is amazing, the impact on the actual UX is much smaller than earlier in the process. But early on, with smaller observational research, we can dramatically impact the UX.

There are a million examples. But just one would be deciding between two types of menu systems: one that is flat and dense or one that is deeper but each part is simpler. Another example is testing an early game tutorial—what content to include and how to show it—before a single line of code is ever written against it.

The point of this model is to show how we can start building confidence early, and then continue throughout the development process. We can start with observing people in the labs, then move to larger studies about what they “like.” And then get some of the “why” to go along with the “what” as user telemetry and other data rolls in when the product hits the marketplace.

How do you package UX?

I’ve presented this model five or six times in the past year or two—to designers, developers, product managers, and many others involved in making games and gaming features. I know it’s not the “right” answer for how to talk about user research, but people have told me they find it helpful, and I hope you do, too.

I’d love to hear more about how you communicate similar concepts and ideas. I think how we as user researchers talk about our work is extremely important to our success. And the more we share with each other, the faster we can up-level our impact and our discipline.

Is evangelizing UX research part of your job? If so, what approach works for you? Let me know in the comments, find my team on Facebook, or tweet us @MicrosoftRI. Want to come work with us? We’re hiring!