Finding the needle in the (data) haystack: 12 useful metrics

There is a lot of talk about data analysis, data mining, business intelligence and similar approaches to systematic improvement. That sounds plausible at first: you can't manage what you can't measure.

But in practice, it's not that easy. And in practice, I come across a surprisingly high number of organizations that do not have a clear understanding of how efficiently they work. Because when it comes to getting specific in the evaluation, it is not so easy to find quantifiable and relevant metrics.

In the following, I describe an approach and also present twelve metrics that help in practice.

Do we have raw data?

A good starting point is an inventory of relevant data. This can take many different forms, for example:

• A Database-based development systemsuch as source code repository (git), ALM tool (Jira), RE tool (Jama), modeling tool (Cameo), etc. These usually have their own reporting functions. In addition, data can often be retrieved directly via an API or dashboard query, or specialized tools can be docked.
• Test, validation and defect data are attractive because they generally make a direct statement about the quality that needs to be improved.
• Customer datasuch as support tickets, telemetry data or the response to marketing campaigns.
• Metrics of the group and department heads. These are often only available in the form of spreadsheets and are then sent by e-mail on a weekly basis. However, these can of course also be evaluated.
• Surveys are also meaningful, even if they provide more qualitative results.

The time component is often missing from the data: After all, we want to improve. It is very laborious (and time-consuming) to derive time-based information from most of the data sources listed. It is therefore best to start immediately to systematically record the raw data on a time-based basis. There are specialized BI systems for this, such as Jama Analyze.

We can only recognize trends if we collect the data at regular intervals.

What do we want to measure?

Even if we have solid, time-based data - what metrics could be derived from it? Here is a small selection for inspiration, from different areas. Many of the metrics can be measured at different levels of detail.

Efficiency metrics

• How much time do we spend in reviews? For example, the time spent in meetings, the effort per hundred elements, etc.
• Efficiency to remove defects - For example, the number of defects found in development divided by defects in development plus defects found in operation.
• Throughput - For example, for sprints, but also for other activities carried out in regular cycles.

Quality metrics

• Test coverage - This metric exists in code (% lines tested), requirements (% requirements tested), and many other areas. It should be noted that 100% coverage is often not useful.
• Defect rate - Quite banal: How many open defects are there? How long do defects normally remain open?
• Escalation rate - for support tickets, for example.

Warning metrics

These metrics can be used as indicators to recognize whether development is getting out of hand.

• Confidence in development - This is often reported via "traffic lights", but this should also be applied over time in order to intervene in time
• Test coverage - The test coverage described above can also show us as a trend whether we will achieve the desired test coverage in time.
• Volatility - This is a sophisticated metric that shows us how much something changes - for example, the number of requirements added, deleted and changed per time interval. High volatility is expected at the start of the project, towards the end of the project this can indicate serious problems, for example the need to change the architecture.

Satisfaction metrics

We are interested in both customer and developer satisfaction.

• Communication in the team - Surveys are a good way of finding out: Does communication work? Do we have too many meetings? Are we using our tools correctly?
• Customer satisfaction - This is often measured directly, especially with software. If this is not done, then it is high time.
• Onboarding success - This is also easier for software to find: How many customers are still using the product after x weeks?

Attention - Do not punish!

For the whole thing to work, it is important not to use the metrics for punishment or "finger pointing"! Incentive systems should also be viewed critically. Because then the metrics are guaranteed to be falsified. Instead, they should be used to select specific measures and then measure their effectiveness: If test coverage does not increase quickly enough, then the tester may need to be released from other activities, or another one brought into the team.

It is also very important to talk openly with the team about the metrics and ensure transparency, otherwise it can quickly lead to uncertainty. That's not good for the team either.

But if data is used correctly and communicated well, the team will be grateful for a functioning feedback mechanism.

Photo by Niklas Hamann on Unsplash