Usability Questionnaires for post-test usability testing

In this article, I will delve into summative usability questionnaires. But first, let me explain the types of usability testing. When discussing usability, it is important to distinguish between the goal and practice of formative and summative usability testing.

Formative Usability Testing

Formative usability testing involves diagnosing problems and making implementations while the product is in development. Typically based on small studies, it is repeated before release. Formative usability testing often uses a think-aloud protocol, where you present users with a number of tasks within scenarios and ask them to share their thoughts (think out loud) while they work through the tasks. The “Wizard of Oz” technique can also be used to test very early prototypes on paper.

Before conducting formative usability testing, it is important to answer the following questions:

• What are the most significant usability issues preventing users from accomplishing their goals or resulting in inefficiencies?
• What aspects of the product work well for the users? What do users find frustrating?
• What are the most common errors or mistakes users are making?
• Are improvements being made from one design iteration to the next?
• What usability issues can you expect to remain after the product is launched?

Summative Usability Testing

Summative usability testing involves establishing a baseline of metrics of effectiveness, efficiency, and satisfaction, or evaluation whether the product meets a set of criteria when the product is nearly finished or finished. This testing generally requires larger numbers for statistical validity. Summative usability testing involves presenting users with tasks or scenarios, as you do in formative usability testing, but you don’t ask them to think out loud, and you don’t engage with them during tasks. This approach allows participants to move through tasks as realistically as possible and provides you with metrics for time on task, completion rates, and other measures. There is compelling rational and empirical support for the practice of iterative formative usability testing, as it appears to be effective in improving both objective and perceived usability.

Before conducting summative usability testing, it is important to answer the following questions:

• Did we meet the usability goals of the project?
• What is the overall usability of our product?
• How does our product compare against the competition?
• Have we made improvements from one product release to the next?

Now that you know the differences between summative and formative usability testing, let’s move on to the standardized usability questionnaire.

What is Standardized Usability Questionnaires?

Standardized usability questionnaires are designed to assess participants’ perceived usability of products during or after usability testing. A standardized questionnaire is designed for repeated use, typically with a specific set of questions presented in a specific order using a specific format, with specific rules for product metrics based on the answers of respondents.

Currently, the most widely used standardized usability questionnaires for assessing the perception of usability at the end of a study (after completing a set of test scenarios) are:

• Questionnaire for User Interaction Satisfaction (QUIS)
• Software Usability Measurement Inventory (SUMI)
• Post-Study System Usability Questionnaire (PSSUQ)
• Software Usability Scale (SUS) — most recommended questionnaire for post-test

Questionnaires intended for administration immediately following the completion of a usability task or test scenario that is part of a larger overall study include:

• The After-Scenario Questionnaire (ASQ)
• Expectation Rating (ER)
• Usability Magnitude Estimation (UME)
• The Single Ease Question (SEQ)- most recommend questionnaires for post-task
• The Subjective Mental Effort Question (SMEQ)- most recommend questionnaires for post-task

The post-task questionnaires will be discussed in my next article.

The QUIS is one of the earliest questionnaires published for post-test to measure user satisfaction with specific aspects of the human-computer interface. It assesses the system, including the overall experience, screen-specific aspects, terminology and system feedback, and learning and help, which contribute to usability. It can be used either in its current form or modified to meet particular research needs. According to the QUIS website, most people use the short version and only the sections that are applicable to the system or product. Higher scores indicate greater satisfaction. QUIS is a widely used questionnaire in user experience research and is particularly useful for evaluating the usability of software systems.

To use the QUIS, it is necessary to license it from the University of Maryland, College Park. At the time of this writing, the paper version fees are $50 for a student license, $200 for a non-profit license, and $750 for a corporate license; the web version fees are $75 for a student license, $300 for a non-profit license, and $1000 for a corporate license.

The SUMI was developed by the Human Factors Research Group (HFRG) at University College Cork in Ireland, led by Jurek Kirakowski. It is a 50-item questionnaire with 5 subscales to measure user perceptions of various aspects of software usability, including Efficiency, Affect, Helpfulness, Control, and Learnability. SUMI is a widely used questionnaire in software usability research. Now the website is unable to access for more information, http://sumi.ucc.ie/.

PSSUQ

The PSSUQ is a questionnaire designed to measure users’ perceived satisfaction with computer systems or applications. The origin of the PSSUQ was an internal IBM project called System Usability Metrics (SUMS) headed by Suzanne Henry, and after content analysis, 18 items remained for the first version of the PSSUQ.

A few rounds of improvements have resulted in PSSUQ Version 3 (16 items), which is the one used today. They are:

• Overall: Average the responses for Items 1 through 16 (all the items).
• System Quality (SysQual): Average Items 1 through 6.
• Information Quality (InfoQual): Average Items 7 through 12.
• Interface Quality (IntQual): Average Items 13 through 15.

The resulting scores can take values between 1 and 7, with lower scores indicating a higher degree of satisfaction.

The System Usability Scale (SUS) has become a popular questionnaire for post-test subjective assessments of usability. It is a questionnaire with ten items, each with five scale steps. The odd-numbered items have a positive tone, while the tone of the even-numbered items is negative.

To calculate the overall SUS score:

• For odd items: subtract one from the user response.
• For even-numbered items: subtract the user responses from 5 (reverse scoring).
• Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40.

Since its initial publication, some researchers have proposed minor changes to the wording of the items. Also, the original SUS items refer to “system,” but substituting the word “website” or “product,” or using the actual website or product name seems to have no effect on the resulting scores. Of course, any of these types of minor substitutions should be consistent across the items.

The questionnaires listed above are the most commonly used ones. One thing to remember is that though these standardized usability questionnaires are general recommendations, all of them have their strengths and weaknesses, and you might find that one of the others is a better fit for your specific situation. Here are some tips:

• Has the product been extensively tested? If yes, the Post-Study System Usability Questionnaire (PSSUQ) is a good tool for fine-tuning an already tested product.
• Are any of the sub-scores for either questionnaire especially interesting or relevant for your research? For example, if you are interested in the learnability of a product, then the SUS is a good choice.
• Is cost a concern? Then choose one of the free questionnaires that do not require a license fee (e.g., SUS, PSSUQ).
• How big is your sample? If you can only recruit a small number of users, it is best to choose a measure that can provide valid results with smaller sizes (e.g., SUS, PSSUQ).

References

• Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann.
• Albert, B., & Tullis, T. (2013). Measuring the user experience: collecting, analyzing, and presenting usability metrics. Newnes.
• https://uxpsychology.substack.com/p/standardized-usability-questionnaires?utm_campaign=post&utm_medium=web

QUIS™ (Questionnaire for User Interaction Satisfaction)

• Chin, J. P., Diehl, V. A., & Norman, K. L. (1988, May). Development of an instrument measuring user satisfaction of the human-computer interface. InProceedings of the SIGCHI conference on Human factors in computing systems (pp. 213–218).
• https://www.umventures.org/technologies/quis%E2%84%A2-questionnaire-user-interaction-satisfaction

Software Usability Measurement Inventory (SUMI)

• Kirakowski, J., Corbett, M., & Sumi, M. (1993). The software usability measurement inventory. Br J Educ Technol, 24(3), 210–2.

Post-Study System Usability Questionnaire (PSSUQ)

• Lewis, J. R. (1995). IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human‐Computer Interaction, 7(1), 57–78.
• Sauro, J., & Lewis, J. R. (2016). Quantifying the user experience: Practical statistics for user research. Morgan Kaufmann.

Software Usability Scale (SUS)

• Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4–7.