11 Essential Software Development Metrics to Ensure On-time Delivery

Posted by Kasey Jones on Mon, Jun 26, 2017

A critical part of running a successful software development team is being able to properly plan for each sprint - especially knowing how many stories or points to include in order to release on schedule. Efficient sprint planning requires a thorough understanding of not just your team's productivity, but its accuracy in predicting that productivity. And because it's hard to foresee every event that might influence your dev team's efficiency, it's critical that you consistently track software development metrics that can indicate when your dev team might not deliver on time. 

Go beyond burndown charts (though we certainly have a soft spot for burndowns) and discover the metrics you need to be tracking and watching to ensure you stay ahead of roadblocks in your team's way. 

We're working on compiling The Essential Guide to Software Development Team Metrics and over the past few weeks, we've been discussing the key metrics you need to be tracking. For our purposes, we segment software development team metrics into four distinct categories: Speed, Accuracy, Quality, and Joy (for more insight into why we choose this framework, read co-founder and Chief Product Officer, Kevin Steigerwald's piece on the topic here). 

For early access to the complete guide, sign up here. 

We've already discussed the 8 essential software development metrics for team productivity, and the 13 essential software development metrics to ensure qualityToday, we're covering the software development accuracy metrics that will help you ensure your dev team is delivering on time. 

~

The Essential Guide to Software Development Accuracy Metrics

Software Development Metrics deliver on time Burndown charts.jpg

Point Commitment Reliability

What is the metric Point Commitment Reliability?

Point Commitment Reliability the ratio of points delivered vs points committed to. This is also referred to as your Say/Do Ratio. Well-functioning organizations build reliable products, and reliable products are the sum of the reliability of the teams building them.

How do you measure Point Commitment Reliability?

Point Commitment Reliability is measured as the points delivered at the end of an iteration or time period divided by the points committed to before starting an iteration or time period. The objective is to measure, at a granular enough level, the value to the customer, so there isn’t any question about delivered scope. Everyone needs to have a shared understanding of what was delivered relative to what was committed.

The units don’t have to be points, either -- this Say:Do Ratio could be calculated based on features or stories delivered, again, as long as everyone is clear on the accepted scope delivered. 

Why do you measure Point Commitment Reliability?

Point Commitment Reliability, or a Say-Do Ratio, is the basis for communicating that your teams are meeting the expectations they are setting. It’s the foundation for demonstrating excellence and predictability on your teams, but, surprisingly, few teams start with this core metric. Trending this over time will allow you to push back on stakeholders that don’t understand how reliable your teams are, and it will help surface teams that are overcommitting, have unclear priorities, or aren’t managing tradeoffs successfully.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

~

Release Window Accuracy

software development metric deliver on time Release Window Accuracy.png

What is the metric Release Window Accuracy?

Release Window Accuracy is the deviation, in days, from when you planned to ship your product.

How do you measure Release Window Accuracy?

Given that you have either a planned date or a window of dates to ship your product, for every day that you’re late, score that as a -1 and for every day that you’re early, score that as a +1. Your current accuracy is the total number of days late or early that you are shipping. If you are very accurate, you’ll have an RWA of 0, but positive numbers are better than negative numbers.

As an example, if our goal is to ship every Wednesday after a sprint, and we end up shipping on Thursday, that would mean an RWA of -1.

Why do you measure Release Window Accuracy?

Shipping is naturally very closely linked to your customers realizing the value of your hard work, and a high performing team that’s not experiencing roadblocks should be able to plan and execute accurately. By tracking your Release Window Accuracy over time, you’ll see if your team is functioning smoothly, or if there’s high volatility in your processes that need attention. It’s important to review RWA in context of other counter metrics to understand the underlying causes of missing commitments.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

~

Feature Adoption

What is the metric Feature Adoption?

Feature Adoption is a custom metric that describes the success of your product feature to deliver value to your customer.

How do you measure Feature Adoption?

Feature Adoption is measured relative to the exposure and subsequent adoption of the product by your users. As a best practice, the success criteria should be defined up front -- during the planning stages of your stories -- with follow up after delivery and launch to compare expected versus observed results.

Awareness or exposure should be decoupled from usage, because users that don’t know about your feature aren’t a good sample to examine for adoption. Also, adoption should be taken in context of your total user base. In many cases, high adoption in a small overall user population may mean that, while highly successful for some users, abandonment of the feature may be the best course of action.

Given that, Feature Adoption can be measured as:

% of users exposed to the feature * % of exposed users having successful outcomes

As an example, let’s say 5000 users see our new feature and click through out of a total active user base of 100000, and then 1000 go on to complete a successful action (desired outcome) we calculate that as:

5000 exposed users / 100000 total users *  1000 / 5000

0.05 * 0.20

1%

Why do you measure Feature Adoption?

It’s important to measure Feature Adoption, because it closes the loop on the build-measure-learn loop, and provides insight on whether the product planning process is working to deliver features that users derive value from. It’s expensive to pursue product development paths with little return, so the faster you can assess whether what was designed, developed, and delivered is delighting your users, the faster you can continue to deliver valuable product. It’s important to resist the temptation to continually move on to the next features without promoting the existing feature set and closing the loop, and to avoid the sprawling cost of maintenance associated with unused product.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

  • Funnel stage 1 for feature (Feature exposure)

~

Net Promoter Score

What is Net Promoter Score (NPS)?

Net Promoter Score is a balanced measure indicating the desire of your users to recommend your product by taking into consideration promoters, passives, and detractors. The Net Promoter (NPS) score ranges from -100 to 100 with -100 being all detractors and 100 being all promoters.   

NPS is a proxy for customer satisfaction.

How do you measure NPS?

Net Promoter Score is the calculated results of surveying your user base with a simple question: “On a scale of 0-10, how likely is it that you would recommend [product name] to your colleagues, friends, and family”.

A detractor is anyone who responds with a 6 or less. A passive is a 7 or 8. A promoter is a responder of 9 or 10.

The percentage of detractors is subtracted from the percentage of promoters, and passives are ignored.

The surveying is done on a periodic basis - typically every 6 months - so you can establish a baseline and trend.

Why do you measure NPS (Net Promoter Score)?

NPS is a simple approach to surveying for customer satisfaction. By establishing a baseline and trend, you and your team can stay out ahead of churn and measure the impact of your customer retention efforts. By keeping the surveying simple, a higher number of responses will mean a more statistically significant result set and more accruate trend.

NPS surveys are conducted periodically, but not very frequently. Consider more frequent NPS-style polling for your product feature releases, asking about specific features, to determine satisfaction with those new features.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

  • Cost-based metrics
  • Referrals
  • Renewal Rate
  • Customer Effort Score (CES)

~

Delivered Customer Value

software development metric on time delivered-value-chart.png

What is Delivered Customer Value?

Delivered Customer Value is a qualitative measure of the value delivered to customers in a given period. For agile teams, this can be an end-of-sprint metric that’s reviewed in the retrospective.

How do you measure Delivered Customer Value?

To measure Customer Value, we recommend using regular team polls to gauge the perception of value delivered to the customer within a given time period. A simple 1-10 scale and single questions: “How much value did we deliver to our customers this period” is a great start.

Why do you measure Delivered Customer Value?

The foundation for success is built on delivering the right products, at the right time, and frequently. If the team doesn’t feel they are delivering value, or if there’s a drop in that sentiment, then it highlights a gap between the product planning process and the realization of the product.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

~

Ticket Churn

What is Ticket Churn?

Ticket Churn is a measure of how many tickets have moved backwards in progress during a period. This is also referred to as Ticket Bouncebacks.

How do you measure Ticket Churn?

Ticket Churn can be extracted directly from your issue and project tracking tool (e.g. JIRA or Pivotal Tracker). Those tools keep track of ticket state changes, and you’re looking for tickets that have transitioned from a started to unstarted state or from a completed to incomplete state.

Ticket Churn is the total count of tickets that have moved backward during the given period.

Why do you measure Ticket Churn?

Ticket Churn is an important metric if your team is struggling to properly scope (right-size) stories or is experiencing other issues stemming from the product planning process. Increasing amounts of tech debt can surface as Ticket Churn as well.

Ticket Churn is a gauge of rework and rework is wasted effort which will impact your team’s velocity and, likely, product quality.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

~

Burndown 

release-bd-chart.png

What is Burndown (also known as a burndown chart)?

Burndown is the visualization of the work remaining to be completed and the pace at which it is currently being completed.

A Burndown Chart usually represents your story burndown or epic burndown.

How do you measure Burndown?

Story Burndown (Feature Burndown) is a daily chart showing the work (stories / features) remaining to be completed and a trendline (based on Velocity) projecting the estimated completion date.

Why do you measure Burndown?

Burndown is measured to understand if the team will complete its work in the originally estimated timeframe. Additional scope, bad requirements, and a team slow down will all affect the pace of completion, and the Burndown Chart is a method for visualizing the impact in order to make course corrections and reset expectations.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

~

Backlog Health

software development metric deliver on time backlog health.png

What is Backlog Health?

Backlog Health is the measure of the readiness of your backlog to be worked on by the team. Backlog Health is calculated as the number of iterations prepared and ready based on the velocity of the team.

We often refer to this as Backlog Burn Rate, because it’s how quickly you are burning through your prepared work as the team takes on new work.

How do you measure Backlog Health?

Backlog Health measures the ready queue depth of your backlog and is calculated as:

Current number of prepared backlog points / average team velocity

If you’re not measuring points, then it can be measured as the number of stories that are ready divided by the team’s velocity in stories.

Why do you measure Backlog Health?

Backlog Health helps you in diagnosing the wellness of your product backlog. A healthy backlog has 2 sprints worth of sprintable stories. Less, and the team will start to struggle with poor requirements and may be idled if they speed up. Too much prepared backlog, and your stories can grow stale and result in a lack of product focus on what’s important to the business.

A healthy backlog relies on story sizing, prioritization, and definition of success for the feature. Tracking your backlog health is key to understanding your product process and to staying out ahead of idle teams, invalid stories, and unhappy developers and product owners.

Other relevant software development accuracy metrics:

You're probably also tracking:

Accuracy counter metrics to consider:

  • Sprint preparedness
  • Story size breakdown

~

Sprint Preparedness

What is Sprint Preparedness?

Sprint Preparedness is the qualitative measure of how prepared you are for the upcoming sprint or were for the past sprint. It’s especially helpful as a data point during retrospectives.

How do you measure Sprint Preparedness?

Sprint Preparedness, or measuring the quality of sprint preparation, is a subjective measure, and we recommend using regular team polls to track trends and stay out ahead of issues. Team polls can be as simple as, on a 1-10 scale, with 10 being the most prepared, “How prepared were you for this past Sprint”. Sprint Preparedness can be measured at the end of an iteration heading into a retrospective, or it can be measured after planning, as you look forward to the upcoming sprint. For teams that don’t follow sprints, you can pick a time period - e.g. bi-weekly - that works for you and your team.

Why do you measure Sprint Preparedness?

It’s valuable to track and trend the team’s take on sprint preparation, because it can be difficult to articulate the need for higher quality stories, or a better breakdown (right-sizing) of stories. Your teams’ success in delivering great product, reliably, is dependent on high quality inputs to their process, so regular polling is a quick and effective way to understand perception and address gaps with your new product process in a constructive way. We believe it lends to a productive conversation at retrospectives, and provides and indication whether a drop in preparedness is a one time event or something larger to be concerned about.

Other relevant software development accuracy metrics: 

You're probably also tracking:

Accuracy counter metrics to consider:

Number of Customer Support Issues

What is the Number of Customer Support Issues?

Customer Support Issues are any issues raised by your customer related back to a feature.

Total Customer Support Issues for a given period represents a trend of the friction encountered by your customers using your product.

How do you measure Customer Support Issues?

The Number of Customer Support Issues is going to be extracted from your support tool like Zendesk or Intercom. Ideally, you’re tagging issues related to specific features to move beyond total volume to drill down into specific “hot” areas.

Why do you measure Customer Support Issues?

By measuring the number of Customer Support Issues, you’re trending the friction encountered by your customers using your product. It’s valuable to review the trend after product releases to determine if new or existing features are generating a large (or larger) support volume. This information can be used by the product team to plan refinements to the product or update documentation to avoid confusion.

Other relevant software development accuracy metrics: 

You're probably also tracking:

Accuracy counter metrics to consider:

~

Did we miss any critical quality software development metrics? Let us know what you think are the most important ones to your team. 

Ready to become the data driven leader your team has been dreaming about?

Get early access to The Essential Guide to Software Development Metrics.

I want early access!

 

Level up your team's impact on company growth. 

Learn the AARRRT KPIs of Product Metrics and become a PM Pirate. 

Download the guide now.