In this #mtpcon London+EMEA session, sponsored by Flo Health, Terry Lee, Flo’s Senior Product Manager, explains how product functions can support an experimentation culture and shares some of the pitfalls that product managers should avoid.

Watch the session in full to see the talk or read on for an overview of his key points:

• An experimentation culture requires people to understand the benefits and company investment into teams and tools
• There are some pitfalls to watch out for. To identify them, you need to identify pillars of growth: speed, learning, improvement, and iteration
• To ensure growth without pitfalls you need to question your data
• A culture of experimentation requires business support. You need multiple product personnel and different perspectives

Experimentation culture

As Terry explains, an experimentation culture is born from two things:

1. The people at the business such as product managers, analysts, designers, or everyone with an understanding of the benefits of experimentation
2. The company making investments into teams and tools

In order to create an experimentation culture, the product team needs to run lots of experiments and have an appetite for risk.

At Flo, the web team ran over 100 experiments in Q3 in 2021. They were able to do this, says Terry, because the team structure supported experimentation and they had the tools and system in place and the trust of others in the company.

Pitfalls

There are, however, some pitfalls to be aware of and, in order to identify pitfalls, the first step is to identify pillars of growth: speed, learning, improvement, and iteration.

Speed

Speed is an essential pillar for any product team as new features and improvements must be delivered quickly. However, with the need for speed, tests are not always thoroughly thought out, there isn’t always proper risk assessment completed or documentation created. Also, directionless speed offers little benefit and can lead product teams down unnecessary rabbit holes.

Learning

Learning is critical for team growth, as you need to learn from mistakes to grow in the right way. However, it’s easy to jump to conclusions and reach false positives based on your learning. Plus, you need to be aware of whether or not your learning is based on tunnel vision.

Improvement

Making improvements on previous learnings is essential to making improvements on the product. However, you need to be wary of if you’re improving the product as a whole. Also, while metrics are helpful, improvement cannot only be defined by metrics.

Iteration

The process of iterating on the product is also an essential requirement for product managers. It requires experimentation and sound thinking. However, iteration of successful tests to production without thought leads to fragmentation. For example, when product managers in large companies are only responsible for one product feature, they might not consider other product features. Also, minor changes can inadvertently affect user expectations.

How product managers can ensure growth

Terry suggests that product managers should question their data to ensure growth and that their product doesn’t become a mismatch of features or a Frankenstein product. Is it on brand? Have you checked other health metrics? Are you creating a false positive? Does this set the right expectations for customers?

Take a step back and ensure your tests are grouped together, the basis for hypotheses, and that changes align with other teams’ work. Constant communication with other user-facing teams is essential, so you must share findings and challenge the norm with your data.

However, careful planning is still necessary, which means accepting reality and planning to avoid pitfalls, plus having an idea of your next steps before kicking off tests.

Business support

Finally, Terry explains, support from the business begins with team structure. Teams need multiple product personnel and different perspectives. It also means access to resources such as design, research and analyst assistance when required. Official team touchpoints such as having a culture of cross-team collaboration and demos and sharing sessions are necessary. Plus, the team needs trust, which means giving teams the freedom to make mistakes and time to plan.

The key takeaways are that creating an experimentation culture is challenging and requires support from the business. Still, there are also some pitfalls of experimentation which product managers must know.

Explore more content like this

Ordinarily, premium #mtpcon video content such as this can only be accessed by Mind the Product members. Today, we’re offering up this talk to everyone in the community. If you like the content and want more like it, you can check out our Mind the Product membership plans.