What is Usability?

When discussing usability and user experience, we refer to how an individual interacts with a website, application, or system. We evaluate their behaviors, attitudes, and preferences. Why do we measure behaviors objectively and attitudes and preferences subjectively? The answer is simple: to assist us, as UX professionals, in improving it. Examples of UX metrics include

• task success rate
• time on task
• number of clicks
• self-reported ratings of frustration or delight
• and the number of visual fixations on a link on a web page, etc.

These metrics can offer invaluable insights into usability and user experience. They can help you establish where you stand relative to your competitors and identify areas where you should focus your improvement efforts. For instance, you can concentrate on areas that users find the most confusing, inefficient, or frustrating. Before exploring usability testing research methods, let’s start with moderated and unmoderated usability testing for you and your UX team.

What is Moderated Usability Testing?

Moderated usability testing involves a moderator, such as a usability specialist or UX professional, being present during the test, instructing the participant, or asking them questions directly. The most common form of moderated usability testing is in-lab testing, which typically involves 5–10 participants and one moderator. The moderator asks the participants to perform tasks within specific scenarios while recording their behavior and preferences toward the questions. In-lab tests are most often used in formative usability testing. The most important metrics to collect include those usability issues, such as issue frequency, type, and severity.

What is Unmoderated Usability Testing?

Unmoderated usability testing is conducted without a moderator, using remote testing technology. Online testing involves a large number of participants simultaneously, making it an excellent way for usability specialists or UX professionals to gain flexibility and collect both qualitative and quantitative data in a relatively short time. Unmoderated usability testing is usually set up similarly to in-lab testing. There are background or screener questions, tasks within specific scenarios, and follow-up questions. Participants go through a predefined script of questions and tasks, and all their data are collected automatically. The most important metrics to collect are performance metrics and self-reported metrics. It may be challenging to collect issues-based data since the moderator isn’t observing participants directly. However, performance and self-reported metrics can help point to issues, and verbatim comments can help infer their causes.

Usability Testing Research Methods for Qualitative Inputs

In-depth interview (IDI)

In-depth interview (IDI) is a qualitative research method typically used to gain a detailed understanding of a participant’s experiences, opinions, and perspectives. It involves a moderator asking open-ended questions and encouraging the participant to elaborate on their responses. The goal of an in-depth interview is to gain a deeper understanding of the participant’s attitudes, behaviors, and motivations. This method is often used in exploratory research or as a supplement to other research methods.

Think-out-loud (TOL)

Think-out-loud (TOL) is a qualitative research method where participants verbalize their thoughts and actions as they complete a task. The goal of TOL is to understand the reasoning behind participants’ actions and decision-making processes. This method is often used in usability testing to identify areas where participants may struggle or encounter confusion.

Think-out-loud (TOL) is a study that is used to get verbal feedback from participants while they interact with your website, app or online prototype.

Questions in a TOL test are crafted in a way to uncover insights into how users behave, make decisions, and think and feel in the context of your product or service.

Retrieved from UserZoom Live Interviews https://www.userzoom.com/live-interviews/.

Usability Testing Research Methods for Quantitative Inputs

Survey

A survey is a research method that involves asking a predetermined set of questions to a group of participants. The goal of a survey is to collect quantitative data about participants’ attitudes, preferences, past experiences, behaviors, and demographics. Surveys can be conducted in a variety of formats, including online, over the phone, or in person. They are often used to gather data from a large number of participants quickly and efficiently.

Net Promoter Score (NPS)

NPS is a metric used to measure customer loyalty and the likelihood towards a company or brand. It asks customers how likely they are to recommend the company or brand to others, on a scale from 0–10. Customers are then categorized as detractors (0–6), passives (7–8), or promoters (9–10), and the Net Promoter Score is calculated by subtracting the percentage of detractors from the percentage of promoters. A higher NPS indicates greater customer loyalty and satisfaction.

System Usability Score (SUS)

SUS is a standardized questionnaire used to measure the perceived usability of a system or product. The questionnaire consists of 10 questions, and participants are asked to rate their agreement with each question on a 5-point Likert scale. The questions pertain to the overall customer experience and product usefulness. Responses range from strongly agree to strongly disagree. The responses are then aggregated to calculate a SUS score, which ranges from 0 to 100. A higher SUS score indicates better usability.

Usability Testing Research Methods for Preference & Impression testing

First-click testing

First-click testing is a research method used to present participants with a screenshot of a page from a website or app, or a section of webpage content, before asking questions. Participants are asked to perform a specific task, and their first click is recorded. According to Sauro (2011), if the first click was correct, users had an 87% chance of completing the action correctly, as opposed to just 46% if the first click was wrong.

This method is used to identify areas where users may have difficulty finding the information they need or completing a task. It can also be used to evaluate the effectiveness of a website or app’s layout and design.

Five-second testing

Five-second testing is a research method commonly used to evaluate the effectiveness of a website or app’s visual design in communicating its intended messages. Participants are shown a screenshot of a webpage or app screen for five seconds, after which they are asked simple questions such as “What is the main purpose of the design?” “Can you recall any particular elements within the design?” and “Who is the website for?”. This method is used to identify areas where users may not be able to quickly and easily identify important information or complete a task. It can also be used to evaluate the effectiveness of a website or app’s layout and design.

According to Hopkin (2006), visitors make snap decisions in just 50 milliseconds to stay or leave your site. Therefore, the five seconds is long enough for a well-designed website to communicate its purpose and main impression. This helps to reduce bounce rates.

I wrote an article about quick test of visual design, which also introduced first-click testing and five-second testing.

Preference testing

Preference testing is a research method used to understand participants’ preferences for specific designs, features, or products. Participants are presented with two or more options at the same time, and asked to choose their preferred option, as well as to explain why they made that choice. This method can be used to evaluate the effectiveness of different designs or features and to identify the most appealing options for users. Preference testing is often used in the early stages of product development to help inform design decisions.

A/B testing

A/B testing is a research method used to compare the effectiveness of two or more variations of a design or feature. Participants are randomly assigned to one of the variations, and their behavior and feedback are compared to determine which variation is more effective. This method can help identify which design or feature is most appealing to users and can be used to inform design decisions. A/B testing is often used in the later stages of product development to optimize the user experience.

Usability Testing Research Methods for Information Architecture

Card sorting

Card sorting is a research method used to uncover users’ mental model and how they organize and categorize information. This helps UX professionals design effective information architecture. Participants are presented with a set of cards, each containing a piece of information, and are asked to group them into categories that make sense to them. Card sorting can inform the navigation and structure of a website or app, and can also identify areas where users may have difficulty understanding how information is organized. There are three types of card sorting: open, closed, and hybrid.

• Open card sort: Participants group cards into categories that make sense to them and label each category in a way that accurately describes the content.
• Closed card sort: Participants sort cards into category groups that are already labeled and defined. This is useful if you’re launching a new page for “watches” and you’re unsure whether to put it under a parent category such as “Accessories” or “Jewelry.”
• Hybrid card sort: This is a mixture of open and closed card sorting. Participants can sort cards into categories you’ve already defined and create their own categories if they feel that your categories are inaccurate.

With more data collected, you can discover the percentage of users who placed a card in a certain category. To validate your card sorting results, you can use tree testing (see below). Card sorting results can lead to improved usability and intuitive information design.

Retrieved from Optimal Workshop OptimalSort https://www.optimalworkshop.com/optimalsort/

Tree testing

Tree testing is a research method used to determine where and why users encounter difficulties while navigating a website or app, and to test the effectiveness of its information architecture. Participants are asked to navigate a text-based representation of the website’s structure without any visual design or branding. They are then prompted to locate a specific item or piece of content within that structure to determine the number of steps required to reach it. This information provides insight into the website’s hierarchy and overall navigational flow. The goal of tree testing is to evaluate the findability and hierarchy of content within a website or app, and to identify areas where users may have difficulty locating information.

This method can inform the design and organization of a website or app’s navigation structure, and can improve the overall user experience. The following measures can be used to evaluate the effectiveness of a website or app’s information architecture:

• Directness: the percentage of users who complete the task without hesitation and find the correct answer on their first attempt
• Success: the percentage of users who complete the task compared to the number of failed attempts
• Time: the time it takes users to complete a task.

By testing, iterating, and validating design decisions, a more intuitive digital experience can be created that is simple for users to navigate.

Retrieved from Optimal Workshop Treejack https://www.optimalworkshop.com/treejack/

Advanced Usability Testing Research Methods

Heuristic Evaluation

Heuristic evaluation is a research method that performed by usability experts evaluating a website or app’s user interface based on a standard set of usability principles, or heuristics. The goal of heuristic evaluation is to identify usability issues and provide recommendations for improvement. The evaluation is typically conducted by a team of three to five experts, who independently evaluate the user interface and then compare their findings. The experts then provide a list of recommendations for improving the user interface based on their evaluations. Heuristic evaluation is often used in the early stages of product development to identify usability issues before testing with actual users.

Learn more about Heuristic Evaluation https://uxplanet.org/how-to-conduct-heuristic-evaluation-85548a355dca.

Live website testing

Live website testing involves testing a website or application in a real-world environment with actual users. This can involve launching the website or application to a limited group of users, such as a beta test, or to the general public. The goal of live website testing is to identify usability issues and gather user feedback on the website or application’s design, functionality, and overall user experience. This method can be particularly useful for evaluating the effectiveness of a website or application’s design and usability in a real-world context, and can help identify areas where further improvements may be necessary.

Eye tracking

Eye tracking is a research method used to track and measure where participants are looking on a webpage or application. Eye tracking can provide valuable insights into how users interact with a design, including which elements they focus on, how quickly they find information, and which areas they may overlook. Eye tracking can be conducted using a variety of technologies, including specialized hardware and software, and can be used to evaluate the effectiveness of a website or application’s layout and design. Eye tracking can also be used to identify areas where users may have difficulty finding information or completing a task, and to inform design decisions aimed at improving the user experience.