In this article, we will define the “bare minimum” set of metrics, talk about product success metrics and discuss what can be considered a true agile measure of success.

Have you ever thought, worried, or maybe, laughed at such statements,such as “You can only manage what you measure”?

Have you ever been asked ,“Can you deliver a project on time, under budget, and within the scope and still be unsuccessful?” and felt very strongly about the answer? 

Have you ever seen a company suspending its business operations in spite of impressive progress in the software development team’s velocity, automated test cover, age, and cutting-edge toolset? 

Or, even,companies still in business and doing, at least OK, with poor code, 20-year-old frameworks, and 99.9% manual test coverage?

If you’re familiar with any of this, then read on…

Here I’ll be offering a condensed review of Scrum.org learning paths through the experience we have gained within our Agile Practice team, with the purpose of answering some of the above questions mentioned and proposing recommendations for efficient use of different kinds of metrics: agile, project, product, leading or lagging. 

But, firstly…

What Are Agile Metrics?

There’s something to bear in mind regarding terminology: the term “agile metrics” in itself isn’t especially helpful - and can even be misleading - as all metrics can arguably be agile, depending on how we use them. However, there are metrics that are so commonly used by agile teams that they’ve gained a “signature” reputation of their own as “agile metrics” - like throughput for Kanban, or release burndown, for Scrum. I, personally, suggest seeing them as only a subset of metrics that can be used by any team or organization.  

In this article, I focus specifically on two classifications: product vs. project and leading vs lagging metrics. 

Should You Focus on Project or Product Metrics?

Spoiler alert! 

We should focus on both :) 

In many cases, offshore service companies just have to assume that the product as a whole and the related product metrics are handled properly by the customer. This means that the vendor team just has to focus on what “it is supposed to do”: implementing the requirements with the appropriate level of quality, on schedule, and within budget. But this assumption is risky and may only be applicable to smaller, more standard, well-defined projects, which address a well-understood business need, where there are no major ambiguities regarding the customer experience and marketing potential, design, and architecture. 

With relatively simple projects ,you can, indeed, do quite well staying in the project-level metrics lane: 

• Features/scope and budget burndowns are helpful for almost any case.
• Measuring both customer and team satisfaction can also be considered an all-around must with both product and project success metrics.
• Would it be helpful to check on the number of defects found during development and after release? Sure.
• Team velocity, throughput, item age, cycle time? The Kanban set of metrics is essential for minimizing waste and maximizing value. 

This one last v-word can be a catch. How do you make sure value is maximized when you push the developers hard to increase the unit test coverage to 100%?

In the case of more complex projects, you should really read the word “project” as “product” and act accordingly. That means living by the agile manifesto principles and collaborating with business people on a daily basis to make sure what the development team is doing is aligned as much as possible with what the product needs.

Product vs Project Mindset

Scrum.org PSPO learning path materials

To really illustrate the importance of this, we shall briefly discuss the Product Management Vacuum and ways to fill it. 

Product Management covers so much ground beyond what is visible for a Scrum team: all kinds of marketing research and focus groups, stakeholders interviews and facilitations, collaborations with UX and UI designers, customer interviews, etc. 

This means that there is always a huge risk that a gap or two can appear somewhere between the Company Vision and the Sprint Backlog. So, in the case of Scrum (though the principle is totally true for any project/product), the whole team must be aware of the necessity (and act accordingly, of course) for constant validation of whether the value is being really delivered according to the vision statement, or that the metrics used by the team are in fact the product success metrics.

Product Management Vacuum

Scrum.org PSPO learning path materials

Once again, it may be that the customer has done all the Product Management homework and hired you only for the purely technical jobs. But it is in your best interest to make sure that the customer’s business is successful, and I recommend you apply agile principles through the Vision-Value-Validation triad which encompasses all of the “short increments-frequent feedback” goodness, as opposed to working off the “classical” assumption that it is possible to plan things the right way upfront, whilst planning a bit of change management just in case ;) 

In plain words, just ask the customer about the product KPIs and goals, then plan and try to align what’s important to the business with what you measure in your team’s work. This is the way to ensure you are optimizing the whole - the product, not just one subsystem. Studies show that trying to improve a sub-system is bad for the rest of the system, so you want to avoid the suboptimizations.

To do this well, I would totally recommend taking a good look at the Evidence Based Management Guide.

Evidence-Based Management Overview

Evidence-Based Management

This framework suggests basing your product measures of success on 4 Key Value Areas and depending on the current lifecycle state of the product (for example, a new and promising startup or a cash cow that is about to retire) select the most relevant product and delivery metrics and use them to adjust the engineering team focus. The Evidence-Based Management guide also contains a wealth of product metrics examples.

In the case of a good money-making product that, at the same time, already reached its marketing potential peak, it may make sense to focus on improving the Installed Version Index in order to prepare a more efficient (and cheaper) support and maintenance phase, than try hard to reduce the lead and cycle time.

And on the contrary, for a new and aspiring product, it may be critical to reduce the Time To Market as much as possible in order to speed up the conversion of the Unrealized Value into Current Value. The ability to Innovate should get a lot of love too.

So, it is not always obvious and intuitive for the engineers - striving for technical perfection and producing state-of-the-art software is not necessarily what will bring money to the customer. As it is also true that a lot of businessmen do not realize that a startup built in a fast and dirty manner has more chances to remain “just another startup” than one built with proper attention to scalability and quality. And managers can help bring some balance and peace between these two extremes, in part, by applying the right combination of metrics and measurements.

So, product and project metrics are supposed to be in alignment and used in combination.

Another Aspect of Measurements: Leading and Lagging Metrics

Another way to make sure your product and delivery metrics are well-balanced and meaningful is by viewing them as leading and lagging. The former can be considered as related to inputs (or pre-product), while the latter measures outputs and outcomes. The former is easier to influence but harder to measure, the latter is easier to measure but hard to influence.

Here’s the illustration used in “The Professional Product Owner”:

“A common example used to illustrate this distinction is trying to lose weight. The ultimate goal is to lower the weight measurement. Weight is a lagging indicator, easy to measure—just step on a scale—but harder to directly influence (unless you cut off a limb). How much you eat and exercise are leading indicators. It certainly is not easy, but you have more influence over these leading indicators. The hope is that as you positively influence calories taken in and calories burned, eventually your lagging weight indicator is affected (hopefully in the right direction). In product development, measuring both is just as important”.

Don, McGreal; Jocham Ralph. The Professional Product Owner: Leveraging Scrum as a Competitive Advantage (pp. 67-68). Pearson Education. Kindle Edition. 

How Do You Make Sure the Metrics Really Work? More on Suboptimization Threat

Still, there are a couple more things to keep in mind about measurements.

Sometimes, you may be tempted to try to incentivize behavior or a practice based on the related metrics. And this has to be done with great caution: it is human nature to try to game any imposed controlling measures - and what can be gamed will be gamed. To mitigate this risk experts advise asking colleagues to help with brainstorming ways to trick a metric before you implement it. 

Also, connecting a metric to material bonuses is very likely to kill the value of measurement as the people will start to work specifically on solely meeting the goal, which may lead to another kind of suboptimization. The metrics will be improving, while the actual performance will be falling per Goodhart’s law. See the illustration below.

Goodhart’s Law

Don, McGreal; Jocham Ralph. The Professional Product Owner: Leveraging Scrum as a Competitive Advantage 

In any case, you should be prepared to review, change or replace metrics periodically, a metric should not be a goal in itself. It is OK for a metric to become obsolete and be replaced.

And last but not least, beware of cargo-cutting a metric: applying it without a true understanding of its nature. For example, many managers who have not yet had a chance to dive deep enough into how agile frameworks work may try to compare the performance of different teams by their velocity, a popular  “agile” metric. This would prove to be very misleading in the circumstances.

Measuring velocity makes sense only for that one team that is doing the estimations, planning sprints, and working on achieving the goals. If your team’s velocity is not growing or even falling, it is an indication that your team’s Product Backlog refinement practices have to be improved. It would be misleading and wrong to expect that different teams should “produce” the same amount of story points, as each team has its own relative scale and measurement. In addition, if managers persist in viewing velocity as a cross-organizational metric, it will lead to suboptimization resulting in teams trying to game the metric. This would also lead to a high level of unnecessary frustration and deteriorating morale.  

Conclusion

So, to recap agile measures of success:

1. Product and project metrics are vital but can be tricky.
2. To be successful in the modern world, businesses should try to adopt ‘product thinking' as opposed to ‘project thinking,' which means using both project and product metrics in combination.
3. The most common problems with the metrics are suboptimization and cargo-culting. 
4. The cure against suboptimization is the alignment of the metrics across the whole system, avoiding focus on sub-systems only.
5. Combining leading and lagging metrics is also recommended. So, are regular reviews, changes, and replacements to the metrics.
6. Cargo-culting can be ruinous and should be avoided by continually educating a team about its pitfalls. 
7. Evidence-Based Management is a great example of a holistic approach to measurements in product development.

As much as it is fun to give a crash course on a complex subject, it is still true that in real life finding the right measures of success is not easy and it is wise to leverage the experience of professionals, alongside one’s own self and team-development initiatives.