To minimize or even avoid bias, ①the participants are the ones who could represent the majority of your target audience，②you ask right research questions，③your sample size is large enough to represent the whole, and ④disclose the true fact when analyzing data.

To recruit the right people, the modes or ways of your invitation, your screening survey/recruiting criteria, and the date and time you invite people should be taken into consideration.

Ⅰ. Bias when recruiting（modes of inviting people and screening criteria）

ⅰ. Recruiting process

About how to motivate people to participate in your research，check this “How to motivate people to participate in your research?”

Sorry that Medium doesn’t allow to insert the sheet. So I pasted the screenshot here. You could also check the pdf. Download it if you need.

ⅱ. The date and time you invite people

Reasons of bias：

●If your company executes a marketing, or publish an Ad, the research result you get after the marketing and Ad may be different from that you get before.

●You will get different answers if you ask people how much they like to purchase gifts before and after Christmas Day. If it is a shopping website, and you choose to publish your questionnaire between 8:00 am to 5:00 pm on workdays, it will be less likely that the majority of your target user will participate in your research because they could not see your invitation.

●For some websites, users will visit your website in a period of a day, or in a period of a week, or in a period of a month, or in a period of a season, or in a period of a year. And the visits will peak in this period.

Solution：

●Before publishing your questionnaire, consider these points:

①When will the users not be busy?

②When will they be active on your website？

③When and in which case will the users be most likely to be motivated to finish your survey?

④When is the date to invite people will best help you gain the most valuable and useful data, with the least bias? Before or after the date？

Ask the right questions(Bias in designing the questions)

Ⅲ. Question design

It is likely to bias the result if your questions are not designed in a correct way.

People are different, their demographics are different, their roles and identities are different, the ways they perceive the world and how they view this world are different. So ask yourself these questions before designing your questionnaire：

●Which kind of information do I want to know? Is this question able to help me gain the information?

●How do people perceive and view your product? What are their experiences of using your product?

●When people see your questions, what do they see and hear at first? In other words, how do they understand your questions, and how do they perceive the information in your questions? Do they understand your questions immediately and be attracted to continue or do they feel it is no business with them so they close your survey?

●What will happen in the next steps? What will the flow look like?

●Do they think your questions are important?

●Are they able to understand these questions?

●Will the order of the questions influence or even bias how people view and their ideas?

●Are people willing to answer your questions? Are they knowledgeable and experienced enough to answer your questions?

ⅰ. On how to ask the right questions, check this story:

How do you know if your research questions are correct?

ⅱ. Solution for designing good questions：

●Design the questions that people are willing to answer.

●Designed questions that people will understand what you want to know, without misunderstanding.

●Design the questions without hypothesis. Rather than asking “Is it because of cheap that you like this product? ”, asking them “How do you think about this product? ”

●Don’t ask abstract questions. Make your questions easy, clear and short. Ask questions by using the words people are speaking.

●Design the questions without terms.

●Designed questions without obscure ideas or words. For example, people have different ideas on happiness or well-being. Even the same person have different ideas on happiness in different time or different stages.

●Design the questions without including two or more concepts. Rather than asking “It contains green and purple. Do you like it? ”, asking “It contains green. Do you like it? ”

●Design the questions without including two negative words.

●Don’t ask people what they will do in the future. Ask them what they have done in recent days.

●Don’t ask the questions that people haven’t experienced. For example, if your participants have not purchased the membership, don’t ask the questions relate to that.

●Don’t ask people leading questions. Girls perform better if they are not being asked about their gender before taking the math exam.

●Design the questions with exit option. For example, “Which color is your favorite?” “Green Red White Blue Other___”. “Other___” is the exit option.

●The options should not overlap with each other. For example, customer service and customer support, New York people and US people, 35 to 45 years old and 45 to 55 years old.

●Don’t ask questions which make people feel uncomfortable or embarrassed. And don’t ask them questions which make them feel their ability is suspected or denied. For example, rather than asking“You are not able to find something to achieve some goal successfully just now, is it because you are not focus enough?” asking “How could we improve our product to fix the problem that happened to you? What result do you want to realize? What’s your expectation on this? ”

●To minimize the bias, you can also ask some follow-up questions to dig deeper.

●Each question contains 3–4 options. The number of the options should not be more than 6.

●The order of the options and questions should be random enough. The purpose is to avoid that people see the former and the latter are associated with each other, or they see the former and the latter as the reasons and result, and these will misleading people.

●If it is a yes/no question, don’t put all the options together. Instead, design the options like the following second one:

●You could highlight some keywords when it’s need.

●People prefer shorter and less questions.

●People could lie sometimes. They may exaggerate or hide some ideas. For example, if you ask their address and tell them you’ll give them a gift, they’ll not lie. If you ask about their income, they may lie. Another reason is because of social interaction, so they’ll answer what they think others are glad to know.

●Ask the real person. For example, you could ask people, “Have Sarah solved the problem for you?” if people mentioned Sarah in the previous conversation. It is helpful for gaining the real data in asking them the real person and event.

●Divide your questionnaire into several pages.

●Let people know how much questions left which needs to be answered in their process of filling the survey.

●Make sure your survey could adapt into different kinds of devices, such as PC(Mac, Windows, Linux), iphone, ipad, android.

●Make sure your question is clear without being understood in different things. For example, about the engagement of using the app: Rather than asking, “How often do you use Facebook every day?” Asking them, “How long do you spend your time on Facebook every day? ” Another metric about engagement is, what actions could be the metrics of measuring their engagement? Check their friends’ posts? Or post and share their own stories or moments? Or other actions?

Ⅳ. Bias from sample size

In general, bias could decrease when sample size is larger. But it requires you more time and effort when the sample size is larger. If it takes a long time for the research, your product team may not wait for that, and your research findings may be outdated.

Check out these two videos about sample size and survey from AonaTalks:

UX research sample size explained: How many users do you REALLY need?

How to be REALLY good at Surveys as a UX Researcher (lots of RESOURCES!)

Ⅴ. Bias in analyzing research findings

●Mistakenly associate the reasons with results while they are not reasons and results for each other. For example, the reason of sunrise is not the rooster’s crow. The reason of using your product all the time is not because people like it.

●What people believe and the facts are mixed. For example, people think or even believe they’ll do something in the future, but it is regarded as a fact.

You could also read some UX books about how to analyze research findings, such as survey data, to help you minimize or avoid bias.

Ⅵ. What other things could you do to handle with bias？

（1）If it is survey, you could ask 2–3 people to be your participants and pilot-test your survey. Then check and make sure your questionnaire and the whole process of the survey works well, make sure the ways you recruit participants enable you with recruiting the right people who could represent the majority of your population, make sure the questions you designed are clear enough and easy to be understood without bringing bias.

What you should do in pilot-test your survey：

●In pilot-test, keep the tools/software, ways of recruiting, ways of collecting responses, ways of analyzing the data are the same as real survey.

●If you couldn’t find the ones who are exactly your target audience, the ones who match the key attributes could be your another choice.

●Observe the process that people are taking your research. If it is a survey, ask them how long it takes them, what problems occur when filling the survey. You could also ask people to share their screen with you, or record the process of filling it, so you could observe that. Then perfect and make sure everything for the research goes well.

（2）How could you handle with that if your participants are not your audience?

●Appreciate people and give them a gift.

●Tell them they are not your audience and cancel the interview or the usability testing, modify your screening criteria and recruit the right people.

●Compare the participants with the right audience, check their differences, and clarify who are in the audience group, who are not.

●Check your screening survey and recruiting process, and change them when it needs.

●Check the date and time of recruiting, and change them if it needs.

Things you have to know:

●Bias is inevitable.

●The thing you have to do at the beginning is to know whether the bias could influence or even determine your research findings. If it is so tiny that it couldn’t lead the product development into a wrong direction, the bias is not worth for much effort and time to be corrected.

If you have any suggestions on this topic, tell me your thoughts!