Lessons learned from Microsoft Azure Data UXR

Coauthors: Samira Jain,

,

Based on lessons learned from developing in-product feedback for Azure Data, we created a framework, dubbed BLUE, for designing and implementing in-product feedback that supports the end-to-end design and implementation process.

In-product feedback is a valuable tool for gaining insight into how users think and feel about a product or service. The key benefit of in-product feed is unlike simulated test environments (e.g. usability), users provide feedback in their live environment while using the product.

Further, common issues that arise from not following a framework for developing in-product feedback include ad hoc feedback mechanisms designed by individual teams, metrics that do not align to business objectives, and inconsistent ways of capturing metrics. In some cases, these inconsistencies can result in disruption to the user experience.

To provide some context to the framework these are three common feedback mechanisms:

1. System-initiated feedback mechanism (pop-up for overall product use)
This mechanism relies on sampling to gather feedback and is generally focused on capturing metrics to better understand overall sentiment using a toast or pop-up message.

Example of a system-initiated in-product feedback mechanism

2. Behavior-initiated feedback mechanism (intercept based on user action)
The user provides feedback about the actions they just completed using this feedback mechanism. An example of this in use would be the question prompt, How helpful or unhelpful was this error message? The corresponding response item used with this question type is a scale from “Not helpful” to “Extremely helpful.”

Example of a behavior-initiated in-product feedback mechanism.

3. User-initiated feedback mechanism (always available)
This feedback mechanism is an ‘always available’ channel for users to provide feedback. This type of feedback is useful for bug reporting and for capturing feature ideas that can be explored through additional user research efforts. It’s important to note, the objective of this feedback mechanism is not to measure satisfaction of a product but rather provide users with a mechanism to be heard.

Example of an user-initiated feedback system.

The Framework

The BLUE Framework guides UX practitioners through the UX health metrics process. It builds on best-in-class frameworks, for example HaTS (Muller & Sedley, 2013) and UX Outcomes (Spool, 2021), and The Customer-Driven Culture (Lowdermik & Hammontree, 2020) to provide direction on:

• What UX metrics to capture and how to capture them
• How to create customized UX health metrics for content
• Which guiding principles to apply to the relationship between Business and UX metrics
• How to embed in-product feedback efforts in the context of real-world collaboration, socialization and the UX research process

The 4 pillars of the BLUE framework.

Build

Build UX metrics. Building metrics is more than thinking about outputs — it’s about getting to UX outcomes, translating the outcomes into objectives, and mapping those to metrics that matter within your product and organizational context.

For example, if the

Organizational goal = Build quality products to increase subscriptions by X%

And therefore,

UX outcome = users should be able to focus on completing their task without the system making it hard for them,

Then,

Objective = capturing users’ perception of how easy it is to for them to use the product

And so,

Metric = Ease of use (Overall, how easy is it to use <product>, 5-point rating)

The aim of the Build pillar is to capture metrics that are tied to organizational and user goals that can be translated into actionable insights later for stakeholders to build products and features that make users’ lives better.

Build principles:

✓ Align with business goals
✓ Investigate product UX outcomes, goals, and metrics
✓ Focus on consistency of scales and ratings

Leverage

Leverage proven mechanisms. We recommend starting with the HaTS framework (learn more) as a guide for the mechanism behaviors, specifically around utilizing random sampling based on individual users instead of page product views. Design the mechanisms such that users can provide both feedback on high-level satisfaction questions and feedback on more granular, task-based questions. In addition to the system-initiated feedback, it’s also important to give users a voice by providing a user-initiated feedback option.

Leverage principles:

✓ Utilize HaTS and adapt as needed
✓ Customize needs around sampling/cadence
✓ Include a user-initiated feedback mechanism

Example of HaTS style granular task-based feedback mechanism

Unlock

Unlock the results. This pillar speaks to supporting stakeholders for how they currently or ideally would like to access the UX health data. After all, they are the users of the metrics, and being user-centered, this helps drive success. The goals for unlocking should make the UX health metrics accessible so stakeholders can empathize with the users’ stories. Planning for the best way to serve up the results ahead of time is an important part of ongoing success and engagement with the data, contributing to a data-driven culture.

Unlock principles:

✓ Make the data accessible
✓ Keep the user story central
✓ Contribute to a data-driven culture

Power BI report showing Ease of Use trends and comments for exploration.

Embed

Embed in practice. The final pillar requires embedding the UX health metrics into your UX research practice and socialization. Showing how we can prioritize the user with these metrics and tracking their impact across the entire UXR process is a powerful story to share. Finally, with unique stakeholders, user goals and flows, there is no one-size-fits all approach. Embedding business data with UX health data helps everyone make better decisions with the user in mind.

Embed principles:

✓ Socialize for understanding so that everyone advocates for users
✓ Combine with UXR process and track impact
✓ Get feedback on the feedback

Example of how in-product feedback fits into a UXR process

Conclusion

We believe this framework provides the guidance necessary for research teams to lead the implementation of UX in-product feedback giving their users a stronger voice. We plan on iterating on this framework as appropriate to evolve our UX health metrics system and continue testing and evolving the BLUE framework along the way. View the full paper on the BLUE framework.