Evidence-based product decisions (Part 2 of 2) - Itamar Gilad - Mind the Product

In this week’s podcast episode, we jump right back into where we left off in part two of this series with experience product coach Itamar Gilad. We carry on discussing evidence-based decisions, and how this approach can enable teams to achieve key product goals.

---

Featured links

Episode transcript

Lily Smith: 0:00

Hey Randy, let’s keep this short and sweet because I need to hear part two of our chat with its Mr. Gilad product management coach for more about the just framework.

Randy Silver: 0:08

No, no jokes today, Lily.

Lily Smith: 0:11

Do you have any jokes?

Randy Silver: 0:14

I mean, you know the quality of my usual jokes.

Lily Smith: 0:19

No comment.

Randy Silver: 0:21

Okay, yeah, so let’s prioritise our chat with Mr over my attempts, and let’s get straight into it.

Lily Smith: 0:34

The product experience is brought to you by mind the product.

Randy Silver: 0:38

Every week, we talk to the best product people from around the globe about how we can improve our practice, and build products that people love.

Lily Smith: 0:45

Because it mind the product.com to catch up on past episodes, and to discover an extensive library of great content and videos,

Randy Silver: 0:53

browse for free, or become a mind that product member to unlock premium articles, unseen videos, AMA’s roundtables, discount store conferences around the world training opportunities. For

Lily Smith: 1:06

mine, the product also offers free product tank meetups in more than 200 cities. And there’s probably

Randy Silver: 1:16

so Okay, let’s move on from ideas to taking it into into market actually. So whether you’re at low confidence or high confidence, you have to then do something about it, you have to do further evaluation or move into more of a build phase and keep testing. You advocate steps what’s the difference between a step and a milestone or anything else that we use in terms of normal sub product development

Itamar Gilad: 1:47

steps are vaguely what people call experiments. I don’t like the word experiments, because I think some of the things we do are not really experiments. I prefer the way statisticians think of experiments is something that is a control element. So we’re sure that the results we’re seeing all relatively show that the results we’re seeing are not just random noise. In some steps, the early steps are about just doing researchers to evaluate and just collecting data, because these things bring evidence, and at the end of every step, we need to come back and reevaluate it. So for me successful step is any activity or project usually a few weeks long, where the end of it, you understand the idea on a better level, and you can re rank the impacts in effort with higher confidence. The thing is to always start with low when you have low confidence in an idea, also low investment. So start out with the simple stuff, with surveys with fake dog tests with the easy things. And only when those also show supporting evidence move forward. Once you build enough confidence, you can move on to the next level, which is about building early versions of the product and putting them in front of customers. So this means usability test. This means all sorts of alphas or early adopter programmes. This means internally testing it through a dog food or fish food, which is a limited dog food just for the team. And once you’re satisfied, these things, also show support, you can move on to the later stages, which are about a much more complete version of the product.

Randy Silver: 3:28

So tomorrow, moving on from ideas, you know, whether you’ve got low confidence or high confidence, you then need to do things about it, you need to experiment you need to put into production and then experiment. What are the steps all of that? How does that work? Well, how are they different than than, say, milestones or other project management type things.

Itamar Gilad: 3:49

So again, our natural tendency is to get excited about the idea and then just go forth, let’s build it as fast as possible that put it into into the hands of the users. And then we will iterate if necessary. And a few things happen. Usually it takes longer to actually complete the project than we expected, sometimes much longer. And throughout this time, we don’t learn anything. So obviously, instead of building the project around engineering milestones or design milestones, we want to build it around learning milestones. And that’s where steps come in steps are short activities or projects that we execute very quickly. In at the end of them, we learn something new about the idea, we validate some assumptions. And a successful step is one where we can risk or the idea we can estimate impact and is with higher confidence. So that’s what we’re trying to do. And then of each step, we can make a decision. Once we risk or the idea whether or not we should pursue this idea or maybe we should park it, maybe we should pivot it and test it again. These are all valid options. And we need to decide how far we want to go along the steps that are very early steps that are just what collecting data and teaching us something about the idea, more advanced steps, which are about building a version of the product and putting it in front of users and measuring the results. And then it is AB experiments and even Holbeck experiments while we launch. And you can go very far with these things. The question constantly is, do we know enough to feel the risk of launching the idea in full is justified. If you don’t ask that all you’re actually testing with 100% of users. So I suggest testing, at least with some users every year. And with risky big ideas, test with a lot of users for a long time. And it might be surprising to many to hear this, but it’s actually faster to build projects this way. There are two reasons for that. One is we waste far less time on bad ideas, because through early tests, often we already can determine that idea is not strong, and we can park it. And other reason is that when we build through short steps, when we put in front of the team, very tangible milestones in three weeks from now, we’re going to test with 12 users, we need this prototype with this level of functionality. In three weeks after that, we’re going to build some sort of MVP version. And that will be tested with more users with a higher level of polish. These are very tangible things, we don’t have time for procrastination for over engineering for scope creep, we’re very focused on the minimum that is still viable to test. And often what we learn is that we can learn launch something much smaller than we first imagine, it still does the job. And that’s perfect, then we can move on to the next idea. So despite what our intuition tells us, it’s much better, much more efficient, to build, measure and learn, instead of just build, build, build.

Lily Smith: 6:59

So when you’re going through that process of the steps, and coming up with ways in which to, I guess, validate or invalidate your ideas, one of the things that I often struggle with or, you know, find difficult to prioritise against is, you know, we talked about impact. And obviously, some ideas are going to have a much bigger impact than other ideas. But do you always, you know, how do you decide which ones to go for? And the sense of the ones with the bigger? Do you just always go for the ones with the biggest impact? Or are you sometimes just optimising? Like, I tend to sort of try and find like a spread of different sized ideas, I guess, in order to make sure that we’re making small incremental optimizations, and then also have those bigger step changes. But I guess if you’re not confident on the impact in the idea stage, how were you then sort of adjusting things in the steps stage?

Itamar Gilad: 8:02

That’s a great question. And this is another case where we really need to rely on our judgement and on good communication. It starts with the goals if the goal is ambitious. And the key result is really forcing us to push yourself outside our comfort zone, we need to find slightly higher impact ideas. But that comes with a trade off, it also means higher risk, because these ideas tend to fail more often than the smaller incremental things. But if that’s really what your manager expects, this is really what the company needs. Let’s go further. Let’s find the big ideas. Even if we’re if they’re low confidence. Let’s let’s optimise for impact. In other cases, we’re more risk averse, we don’t want to shake the boat too much. So in this case, we will go for the small things or the things that all safer. And in some cases, we just need to load balance, also our resources. So we have a limited number of people, we want to give an idea to each one or two couple people jump on an idea together. So we’re also balancing based on that. So there’s no one way it’s more of an art than a science. But the thing these are the key factors, the risk, your risk appetite, your available resources, and also which ideas are variable, because sometimes you don’t get to choose you just have a predefined set of ideas to work with.

Randy Silver: 9:28

How do we know when we’re done with steps? Are they a Stage Gate process? Are they an organising principle? Are we ever finished? Or is there a defined end to the steps?

Itamar Gilad: 9:41

For my steps, there’s just a way to structure your project around learning. And the last step, by the way is the delivery step. It’s the most important one and in it there might be multiple classic engineering and design milestones. But I would argue that before that you want to do this kind of B build, measure learn steps, in some people call this discovery this phase. And then the other phase delivery. I’m of the camp that argues that during discovery, you’re already developing your there’s some what, and when you move into delivery, you’re already have some version of it sometimes pretty evolved mean viable product, alpha, whatever it is. And even during the delivery, you keep learning things. And sometimes you start iterating, and going back and changing things. So there’s some learning happening there as well. So whether you call them steps or build measure, learn loops, or discovery versus delivery, it doesn’t really matter. It’s all the same thing to me. And

Lily Smith: 10:39

how do you factor time into all of this as in? Are you like committing a certain amount of time to each step in order to get to confidence or to make a decision? Or is it just, you know, I’m assuming it’s not infinite time until you feel like yes, we’re gonna deploy this or no, we’re gonna commit

Itamar Gilad: 11:02

absolutely to the time box. Part of the problem that I see is that we’re kind of working on two different levels of planning, we have the project, which sometimes is shorter, just a few weeks, but sometimes it’s months or even quarters long. And then we have the Sprint’s or the, the iterations, which are very, very short. And connecting the two, sometimes it’s not very trivial. So we don’t see a lot of meaning in each sprint, because it’s, they’re just small increments towards the bigger project. Steps create kind of a middle ground, the maximum a few weeks long. And in this few weeks, we want to both build the version, launch it, run the test, collect the data, analyse it, so it’s it’s pretty ambitious. And it’s a project. Once we start involving engineers, designers, it is a project a mini project with a very concrete target audience with very concrete target date with very concrete scope. And it can map into a few sprints, or it can be a sub part of a sprint, it doesn’t really matter as long as we are able to deliver. And just having this smaller context really brings life to the project. Because I don’t know if you’ve ever worked on one of these not launching anything for 12 months projects. It’s really a different dynamic once you you work with steps. And again, build measure, learn loops or discovery, whatever you want to call them steps is not an original invention of mine. It’s just a different way to build projects.

Lily Smith: 12:32

So technically, then, we are breaking our product development down into mini projects. We’re all project managers.

Itamar Gilad: 12:45

Yeah, I realised that the word project is very low.

Lily Smith: 12:57

Randy, do you fancy levelling up your product management skills?

Randy Silver: 13:02

Always?

Lily Smith: 13:03

Are you ready to take that next step in your product career?

Randy Silver: 13:07

I thought you’d never ask. Well, you’re in luck.

Lily Smith: 13:11

Mine, the product runs regular interactive remote workshops, where you can dedicate two half days to honing your product management craft with a small group of peers. You’ll be coached through your product challenges by an expert trainer, and walk away with frameworks and tools you can use right away.

Randy Silver: 13:29

And they’re offering a one time 15% discount for any of their August 2022 classes taking place on August 11, and 12.

Lily Smith: 13:38

You can choose from product management, foundations, communication and alignment, or metrics for product managers.

Randy Silver: 13:46

Just use the code summer 15. When buying your ticket, that’s su M M er one five,

Lily Smith: 13:53

find out more and book your place in an August workshop at mine the product.com forward slash workshops.

Randy Silver: 14:05

As someone who has been part of some of those year long, no launching anything things in my past, I want to hear a little bit more about what that transformation is like what happens to teams, how do they how do they respond to this? How do they embrace it?

Itamar Gilad: 14:22

When I teach this to engineering teams, they are the biggest supporters because they are in a situation where they keep getting these plans, dropping on them from the top sometimes from the product manager for sometimes from even the product manager doesn’t know why we’re building what we’re building. And they’re just there to deliver. And most engineers don’t love this mode of work, they want to be involved. And this system kind of gives them the opportunity to do this. They can be involved in setting the goals some goals can come from the engineering teams or some candidate goals. Of course the managers the stakeholders are involved in influencing the goals, but definitely the team shouldn’t be either to, they can propose ideas now that we’re opening the stage for more people to suggest ideas, and we’re testing more ideas. And it’s not just this competition of who wins differently, someone from the team can come up with an idea. And because we’re much, much faster now this idea can come to life within a couple of weeks, it’s much more rewarding. And then the steps create, as we said, a sense of immediacy and more excitement. And every time we get results in, the customers love it, the team gets excited, then it’s mostly about connecting this to the reality of Scrum or Kanban, that they use, JIRA and all this good stuff. And for that, I suggest a tool I called the disc board, which is basically either a physical board with sticky notes or a digital board. It has three columns, essentially one for the key results that we’re trying to, to follow. And by the way, this could be also technical key results. I don’t like having hidden projects where, yes, we have our business goals, but also 40% of our time is going to some hidden project. Absolutely the lids of the team should agree in advance to this quarter, we’re also devoting some time to killing technical debt, or improving the infrastructure or doing killing some design that these are all good things. So the juice board reflects all the goals the team has, and ideally not not more than four key results. Next to it, the current ideas, we’re testing for each one of these key results, some key results may wait and some key results are active. And we have a bunch of ideas. And next to them the immediate next few steps for each one. And my suggestion is that the leads will manage the board will update it because they’re going to be a lot of changes constantly. But the team will meet on a regular basis, usually before the planification or the sprint planning and look at the board and an update, what’s the status, what’s preventing them from moving on the steps, etcetera, it’s a great opportunity. It’s also the checkpoints to follow the OKRs. Because we know that OKRs will not happen unless they’re regular checkpoints. It’s also an opportunity to reevaluate if we’re following the right ideas. So within half an hour, you can do this whole kind of mini status meeting that is incredibly important to create the context. And to remind everyone, we’re not just working here to move tickets to the dance state, we’re trying to achieve something bigger. This context really helps the developers that have been told this many times, because they now know, they’re not just implementing a small change, they’re actually connecting it to a step. And that step is validating an idea. And that idea is connected to a goal. It also works in the opposite direction. By the way, the managers and the stakeholders are often very frustrated with the team because the team keeps iterating and churning out stuff and doing demos. But nothing comes to the other end, it’s it seems very slow and tedious to demand all this churning adult. And they keep waiting there and saying, Well, when are we launching this feature that I’m waiting for? When it does launch, sometimes very disappointed, because it’s not exactly what they expected. So with an evidence guided system, like just, they have a lot more visibility into the ideas the team is evaluating, they can see where their idea is taking up and how it’s being tested, they can see the steps. And usually they’re pretty interested in the results. So I recommend to PMS to share the results and even to involve stakeholders and managers in evaluating what they mean. The tasks not so important that you know the stuff that lives in JIRA, the work items, etc. They don’t care about this. But this helps them build trust in the team. Because this, they see that the team is following the process. It’s very transparent, it’s very objective. And they see how this connects to ideally, to business results. So over time, with enough cycles, we start seeing more and more trust being delegated or more ability to delegate to the team to make product decisions.

Lily Smith: 19:00

And you’ve mentioned that some of the steps can sometimes be business modelling, like financial modelling, or user research. So when you’re planning your chessboard, like, you’ve kind of talked about it a lot in the context of planning work for a development team. That is it work better when it’s more of a cross functional team that has all of those kinds of capabilities within it?

Itamar Gilad: 19:25

Absolutely. I mean, if the first step is someone needs to do data analysis, and that someone is either we’re lucky we have a data analyst or data scientists or it’s the PM, you know, the person who fills up the gaps. So absolutely put a sticky note with the name of the pm there if you want to put owners on the sticky notes and say, analyse the onboarding experience and see how many people we lose in every step of the onboarding. Absolutely, it’s step number one. We can run other steps in parallel by the way, it doesn’t have to be sequential. So I I strongly advise to do a cross functional chessboard not just keep the work of the pm or the designer hidden from the team. And often, the team can help in things that are usually outside of the scope of responsibility. They can help run experiments, they can take notes, they can analyse data, they can do a lot of stuff. And they sometimes are very happy to do these things. And it’s much more efficient if they do them. Because someone needs to do this. If you start being evidence driven, you start finding bottlenecks in PM, and classic pm and designer work, it’s really nice if the team can step up and help with these things.

Lily Smith: 20:40

I think that’s a really good point. And something that I always try to encourage in terms of a mentality and my team is like everyone’s a product manager. And I think this either, as we try and work in this way of like, beginning to get evidence or trying to get evidence for things before we start to build them, it means that there tends to be what I find there tends to be a bit more work upfront, before you start building. So then having one developer or one product manager and six developers, the balance, there isn’t quite right always when you have to do a bit of financial modelling or some data analysis or user research, and you’re getting the team involved in those activities.

Itamar Gilad: 21:28

There’s another side to this PMS tend to be very busy also when they work with teams, because there’s an expectation for them to create the perfect ticket with full user stories with fine detail and definitions and data, etc. And it consumes a lot of time. If we use this model of steps, and we give the team a lot of context, they understand the problem we’re trying to solve, we’re trying to build smaller things just for testing. A lot of times you don’t need to give that much detail, you can actually delegate some of the decisions to them. When I worked at Google, I was perfectly happy when an engineer would invite me to their desk and show me something they cooked up with a designer and asked me what do you think, and I didn’t have to tell them and they came up with ideas that were far superior to mine, I would never have been able to come up with those. And it will take me a long time to turn them into a specification. So that’s what you want to do. And I think in general, throughout my time there, maybe 50% of what was built was actually my idea or my own mind specification, it was kind of so by creating the context and by giving people the chance to contribute. While stay in the loop don’t just delegate all the decisions, you’re perfectly capable to do more of the upfront work, because you kind of are deputising them in a sense to help you do your your classic work.

Lily Smith: 22:49

So what kind of teams have adopted this approach that you’ve worked with? Are they generally kind of starting from scratch or just having a review of the ways of working or are they kind of already in a good place and then just using this to finesse the way that they’re working and, and what has the impacts then of implementing this,

Itamar Gilad: 23:12

the most common case I face is an existing organisation that has already a product organisation. And they want to improve, they want to reread the books, they want to move into this world of discovery and to this world of lean startup. And they’re struggling with it either for cultural reasons, or because they don’t have the tools. Sometimes the product managers are actually X project managers or UX designers to do like the background. And the really wild understand the principles, they’re just looking for a framework to put it all together. So that’s usually the classic case for just, and I worked with teams of all types, business, to consumer business, to business, platform and system to use medical, even companies building tractors or equipment for tractors, all sorts of things, you need to adjust the system. It’s not a one size fit all. Not everything is necessarily perfect for every company. But that’s a matter of trying in any iterating. So a process just like a product is something you need to be a bit kind of cautious about. Give it a chance experiment, validated actually moving you in the direction you want and then change it all iterate on it. That’s the way I recommend doing it. What are the results? So usually one immediate result is that people understand they’re not very good with metrics right now they don’t measure the right things they’re OKRs are usually the realise are not as good as they should be. But on a more positive note the discussion the level of discussion usually improves. Because once we introduce ice into the discussion, a lot of these kinds of opinion based argumentative sort of this discussion, the battles of opinions gets pushed to the side. And we and the companies start being more concrete and more determined, I would say about launching ideas of high impact long term, I would love to tell you that I have extensive research about business results, etc. But I don’t think that actually exists even for some of all the things. But anecdotally, I did see companies that really benefited from adopting at least partially the framework. And it’s absolutely not just about just just it’s about embracing also design thinking and research and data analysis and product discovery, just as just a metaphor framework that kind of puts these very deep ideas into context.

Randy Silver: 25:50

I don’t know anymore. It sounds like you need to go through a few more steps to up the scale if we’re just stays on the confidence metre. Yeah, that’s Sorry, I had to I had to. That’s just one last question, because we’ve kept you for way past anybody’s bedtime on this one, so And thank you for sticking with us. But last question, for anyone who’s trying to implement this for the first time to bring it to their company, or even to their team, any top tips for the first time you go through this for a successful implementation.

Itamar Gilad: 26:27

First off, just to plug my own content, you can subscribe to my newsletter, and you’ll get an automatic email from me with links to basically everything I mentioned, including a number of ebooks, where I shared the system, and blogs and confidence metre as a downloadable calculator and other stuff. This is the end of the commercial plug. Second, I suggest you start by analysing where the weakest thought is right now, is it’s really hard for you to create goals, you create too much output goals, it takes forever to do the quarterly planning starts with goals. Start with the goal layer. If you’re really struggling with ideas, you’re launching a lot of bad ideas, the discussion which ideas to launch is taking too long. Start with IDEA level, introduce ice, start collecting ideas in an idea bags and try to allow the team to iterate on more than one winner idea, so to speak. If you are rushing to launch and you’re not testing along the way try to introduce steps to try to maybe introduce something like the juice board where we next to each idea we ask yourself, What’s the next step? What are the biggest assumption? And what are the things we need to validate in order to gain more confidence about the idea and try to wait right if this is hard, because some teams are not set up for this, especially large b2b, that’s a big transition. But it is a worthy goal. And this is something you can put into your OKRs as well, some of these things. So my top tip is find or your biggest pain point is and start implementing it there. Tip number two, by the way, is make the case to the executive team and find the champion. So convince someone that this is really the most impactful thing you can do for the business or one of the most impactful things and have that person support your initiatives within the organisation?

Randy Silver: 28:29

Is it better to take this to them before you implement or to start implementing small with the team and then share some results and take it to them that way? We’re losing

Itamar Gilad: 28:39

depends on the organisation. I think a grassroots movement where we just do this internally within the team and then we while we are with the results sometimes works. But if they don’t know what you’re talking about, and they still expect you to work in the old way, it’s sometimes blows up in your face. I think it’s worthwhile to have the discussion upfront. But come prepared. Analyse the roadmap from the past years analyse business performance, try to put $1 number or euro number of how much time we’re wasting right now. Implementing things that don’t work and make the case why we need change and then propose the change and try to make it gradual. Try to create a safe sandbox. Maybe a couple of teams will try to work this way for a couple of quarters, and then we’ll analyse the results, etc. Try to gradually build their trust without trust with if it’s completely top down, then you’re facing an uphill battle so you really need them on your side.

Randy Silver: 29:43

It Amar, thank you so much. It’s been a lot of fun talking to you about all this today.

Itamar Gilad: 29:48

The pleasure was all mine. Thank you for inviting me.

Lily Smith: 29:51

Thank you The product experience is the first and the best podcast from mine the product. Our hosts are me, Lily Smith and me Randy silver. Louron Pratt is our producer and Luke Smith is our editor.

Randy Silver: 30:17

Our theme music is from Hamburg baseband pow. That’s P AU. Thanks to Arnie killer who curates both product tank and MTP engage in Hamburg and who also plays bass in the band for letting us use their music. You can connect with your local product community via product tank, regular free meetups in over 200 cities worldwide.

Lily Smith: 30:38

If there’s not one near you, maybe you should think about starting one. To find out more go to mind the product.com forward slash product Thank