We’ve all done it. We say something to make us seem like we fit in. Maybe we want to look cool, or perhaps we just don’t want to stick out.

Someone once told me that pineapple on pizza is the most disgusting and horrible thing they have ever eaten. I told them that I think it’s alright. In reality, a pepperoni and pineapple pizza is my favorite go-to.

Whatever the case may be, what we say isn’t always accurate. “People bend the truth to be closer to what they think you want to hear or what’s socially acceptable.”

These little inaccuracies don’t usually matter too much on a day-to-day basis. Telling someone that you think pineapple is alright isn’t something crazy that will come back to haunt you. These little inconsistencies, however, can turn research and projects upside down.

You might interview a user who tells you that they love your product. In reality, though, they wouldn’t actually buy it. In the moment, the product seems great, but when it comes down to it, they wouldn’t spend the money.

Differences between what we say and what we do can come in many forms, both intentional and unintentional. Let’s take a look at a few.

The Hawthorne Effect

The Hawthorne effect occurs when someone changes their behavior because they are being observed.

For example, let’s look at a study done on The Sims. The Sims is a life simulation video game that allows people to do things like build a house, make characters, get a job, find a virtual spouse, among tons of other things.

Rod Humble, former Sims studio head, led a focus group to see how a bunch of different people played the game. They first watched people play the game, then brought them into a room with others to discuss.

“I remember a bunch of young guys,” said Humble, “and they get into the room, it’s a mixed room, and we’re like, ‘Hey, what did you do?’ and they’re like, ‘Murdered people. Went in and starved people, had sex with everybody in the town.’ But actually, what you did is you redecorated that bathroom, right? Like, that’s actually what you did. There’s this idea that there are some things you should say you’re doing, but actually, no, you’re cooking, you’re making house.” — Rod Humble

In order to seem cool, manly, or whatever they were going for, these guys talked about doing cool things, like sex!

In reality, they made their bathroom look pretty.

The Hawthorne doesn’t just play a part in what we say — it plays a part in what we do.

There was a study done among medical staff at a hospital in Berlin. They wanted to see if hand-washing compliance would change based on whether or not the staff was aware they were being observed. Researchers found that when the staff knew they were being watched, there was a 55% increase in hand-washing compliance.

There are a number of different reasons why the Hawthorne effect might occur in a study. Whatever the case may be, it is important to try and mitigate the effect in order to get valuable research information.

Preventing Hawthorne Effect

There are a few different methods we can implement to try and get genuine data when observing test participants.

1. Unmoderated testing - There is no person there observing the testing. This includes in person and over a call/video call. Participants will be able to answer any questions or do any activities at their own pace. The pressure of being watched is relieved. (That being said, simply knowing that you are participating in a study can alter behavior.)

I know we can’t always do unmoderated testing. Sometimes we need to ask the user questions to get a better understanding of the “why”. For usability testing in particular, having the participant speak aloud and talk through their process can give invaluable information. Sometimes, however, there is no “why” that someone can give. We don’t always know why we do things. We might think we know… but we really don’t.

“Take for example the stocking study by Nisbett and Wilson in 1977. Customers reported on which socks they preferred out of a lineup; A, B, C, or D, from left to right. When asked why they chose what they chose, they had a variety of reasons, but a good deal of the participants chose D, or the furthest set to the right.

Then the researchers changed the order and repeated the experiment. These stockings were of IDENTICAL quality, and, yet again, independent of which pair was in the “D” position, participants overwhelmingly chose D.

Come to find out, its not because a pair were ever any better or wose [sic], but because most people are right-handed and its easier to point to D, than any other pair.”

— Nick Lawrence in a comment on his Medium article

2. Keep your hypothesis to yourself - Sometimes researchers might give participants clues or hints to what they expect or hope to find during the study. This can be as simple as asking leading questions or looking surprised when the participant does something. This can result in users changing their behavior to align with what the researcher thinks or wants to happen. Stay mysterious and unbiased!

3. Make it anonymous - Keeping everything anonymous takes off the pressure to bend the truth or try and fit in. No name, no photo, no worries. Someone probably won’t want to admit they didn’t wash their hands after going to the bathroom when other people might judge them for it or think they’re gross.

Users don’t know what they want

“A lot of times, people don’t know what they want until you show it to them.” — Steve Jobs

What we say we want, isn’t always what we want. That’s just how people are. A lot of the time, people will say that they really want something. That thing, however, might not be the best option. It’s like going to a restaurant and wishing that you ordered what your friend did when the food arrives.

Say, for example, that 50% of survey respondents claim they would buy more from ecommerce sites that offer 3-D product views. Does this mean you should rush to implement 3-D on your site? No. It means that “3-D” sounds cool. The world is littered with failed businesses that banked on people’s attitude toward hypothetical products and services. In speculative surveys, people are simply guessing how they might act or which features they’ll like; it doesn’t meant [sic] they’ll actually use or like them in real life.

— NN/g

Users might say they would love 3-D product views. Thousands of dollar spent implementing this new feature later... and sales haven’t gone up. Money and time down the drain.

The user is supposed to tell you their problems, but not the solution. It is our job to figure out what they really want and what they really mean.

Real-world example

Shoes of Prey, an Australian fashion startup, eventually went out of business because they listened to their customers.

After some various interviews and talking to their customer base, researchers found that customers wanted to customize and purchase their own shoes.

Then they proceeded to use up all of their time and resources to create something that customers didn’t want to use. With their findings, they prepped to scale into $100M’s of revenue. Instead, they had to shut down the entire business.

“We learnt the hard way that mass-market customers don’t want to create, they want to be inspired and shown what to wear. They want to see the latest trends, what celebrities and Instagram influencers are wearing and they want to wear exactly that — both the style and the brand,” Fox said.

— Smart Company

No, the customers weren’t lying to the researchers, “it’s just they can’t tell you the truth because humans are faulty forecasters.”

They could see their future selves customizing shoes, sure. That sounds wonderful! It’s what they have always dreamed about.

In reality, they would rather browse. They don’t want to waste time and money on custom shoes.

Memories are flawed

Interview or survey questions that require participants to recall information aren’t the best bet.

If you ask a user their process for buying clothes online, what they tell you might be different from what they do. They might remember incorrectly or leave out important details. Maybe they missed an important button or UI feature, but they can’t tell you because they didn't see it.

In addition, users might add in their own reasoning for why they acted a certain way or did a certain task. This can add in potentially untrue bias for their behavior.

“In reporting what they do remember, people rationalize their behavior. Countless times I have heard statements like ‘I would have seen the button if it had been bigger.’ Maybe. All we know is that the user didn’t see the button.”

— NN/g

Someone might say that they would have seen the button if it was bigger, but we would only know that to be true if we observed it. Let’s say we do make the button bigger. If users start to find it then, yes, that was probably the issue. If they don’t, maybe the button has bad contrast or is placed in a bad location. We won’t know for sure until we see it.

Further Reading

• GreatBrook — Conformity Bias →
  Discusses how people will change their survey answers to conform with the norm. They also give great tips on how to avoid conformity bias.
• Mitigating the “Hawthorne Effect” in Simulation Studies →
  Research paper on the potential impact of the Hawthorne effect on simulation studies.
• NN/g — First Rule of Usability? Don’t Listen to Users →
  When and how to listen to users.
• Practical Psychology — Hawthorne Effect →
  Some reasons as to why the Hawthorne effect occurs and some real world examples.
• SmartCompany — Why did Shoes of Prey fail? Because it listened to customers →
  Discusses how people are faulty forecasters and gives advice on what you should do to get accurate data.
• Toptal — Don’t Listen To Customers →
  Goes over the science of how we think in “two systems” and why this makes it hard for people to know what they really want or mean.

You might interview a user who tells you that they love your product. In reality, though, they wouldn’t actually buy it. In the moment the product seems great, but when it comes down to it, they wouldn’t spend the money.

Until the money is handed over and the customers are smiling, don’t trust what your users say. Look at what they do.

If you have any thoughts or experiences to add, let me know! Respond to this post or let me know on LinkedIn! I’m happy to talk about anything UX related.