software testing looking for a book in the library

The QA Metrics – ways to communicate it

Everyone from the industry wants to deploy into production, weekly, daily, and even hourly. Current software development is fast-paced and this requires reliable and predictable quality gates. Testing on its own doesn't cover the need to improve on software quality. When evaluating software, it's mandatory to use efficient test metrics and to adapt the way we communicate about quality. With this article I try to emphasize how important it is to share information about product quality in a clear way and to use the right metrics to measure the level of quality.

Author: Iulian Mitrea, Quality Assurance Discipline Coordinator

Iulian Mitrea

Quality Assurance Discipline Coordinator

In the last couple of months, I’ve experienced quite a few situations where it was needed to communicate to different stakeholders’ various information about the quality level of a certain software or piece of functionality.

One of the greatest challenges in software testing lies not in the actual testing process or generating value from it, but rather in finding the best way to share the significance of the testing. Understanding what to communicate, when to communicate, who to communicate with and how to communicate are vital skills that every tester should master at some point.

In general, the value of testing could be summarized in 3 simple areas:

Bugs, Test Coverage (Functional & Non-Functional) and Quality or Risk Assessment

To make myself clear, here are some examples on how we can communicate the testing value:

Good examples:

  1. Bugs: “There’s one blocking bug in the newsletter flow but take into account this happens only for the new marketing campaign”. After this message is shared let the team decide what is the best action that can be taken.
  2. Test Coverage: “You asked about the performance of the payment functionality? We don’t have a lot of performance tests to cover this area, but I can say that this is our prio number 1 for this sprint.”
  3. Risk Assessment. “With this release we ship more risks than actual value, I recommend delaying the release day, but I’m OK if you consider that are reasons to ship it as it is.”

Bad examples:

  1. Bugs: “The situation is like this: We have 123 Active, 25 Resolved and 5 bugs on New. The time we spent solving a bug is going up. The total number of bugs is increasing.”
  2. Test Coverage: “We have 93% execution rate for the tests from this week. More tests will be added next week and this will bring us to 70% code and 80% feature coverage”
  3. Risk Assessment. “This release shouldn’t be validated. It adds too much risk to the business and we aren’t sure how the production will handle.”

Metrics should be the main resource that we think of when there’s a need to share relevant information with different types of audience. Metrics have universal applicability and working with it makes it a universal skill for anyone seeking to achieve and demonstrate meaningful results.

Let’s try to explore together the significance of Quality Assurance metrics, understanding their various nuances, how to efficiently implement them but more importantly how to communicate this information to those who need it.

1.Create metrics specific to your project & team context

Align your metrics with the core functionalities and the main goal of the product you are working with. Having this in place helps you to target your effort and to enhance areas of the software that matter to its user satisfaction.

Customized metrics will help the QA teams to direct their focus on what is truly the value of the software and at the same time make sure that the most critical areas for the user are continuously elevated. In the same time metrics will be influenced by team structure, expertise and workflow. A team where most of the colleagues are specialized in automation testing might focus on metrics like code coverage or the percentage of tests automated. On the other hand, a team that is transitioning to automation could have different metrics prioritization, such as time spent to convert manual tests into automation one.

Keep in mind: Customizing QA metrics to your product and team context is about measuring the right aspects of software quality. By laboriously selecting the right metric and continuously refining them you make sure that the QA efforts are correctly coordinated with your products and team’s needs. 

2.Incorporate metrics into routine process

To engage in a culture where metrics guide continuous improvement there’s a need to empower teams in using metrics into their routine.

Each team member of the product team needs to be encouraged to engage with QA metrics. This activity could consist of involving developers in building/using relevant code quality metrics or aligning with UI/UX colleagues on user experience metrics. Getting into a routine could be as simple as starting the sprint with metrics overview and then incorporating metric discussions in stand-up meetings. It requires a regular involvement with metrics to keep them in the team’s mind.

It's important of course to maintain a continuous improvement cycle. The team should regularly bring feedback about the effectiveness of certain metrics or the process itself. Having such a routine ensures that defined metrics remain relevant, practical and aligned with the team’s and software solution’s evolving needs.

Keep in mind: All the effort involved in the metrics subject should be linked to a broader team and organizational goals. This alignment helps everyone to understand how this effort contributes to a broader success rate in the company.

3.Drawbacks in the usage of QA metrics

If we use some basic examples, we can agree that a high number of bugs in a new feature should create panic during early development stages. However, the same number of bugs in mature product should become a red flag. The point here is that we need to have a close look over the context behind any metrics. This will help everyone to have an accurate interpretation and response.

Don’t burden the QA reporting with overcomplicated dashboards. If you or your team are spending more time tracking & updating metrics than improving the overall quality the reporting system needs simplification. Reports about quality should help, not obstruct the QA process.

Metrics should help us to identify where improvements are needed. This doesn’t mean that we can understand why issues are occurring. To understand the root causes, use open communication and collaboration withing the team. Don’t fully rely on metrics to reveal what team communication can easily do.

A large variety of metrics are usable by different people for different purposes. Identifying which information is helpful and worth sharing, it is crucial for test professionals. By understanding the value we can get from different metrics, we can take informed decisions that contribute to the overall success of our product. Personally, I get the most information from the metrics below:

Test Quality  

  1. Defect leakage: With this one we measure the defects which are found in a production environment after the team has officially released the software.
  2. Defect Spillover: Measures the defects that don’t get resolved in each iteration, meaning they carry over to future iterations.
  3. Defect Density: This counts the number of defects found per thousand lines of code. By viewing the defect density on software modules, you get insight into the weakest areas of the applications you build. You can then decide whether those areas with high defect densities should be released at all or made simpler, so they have fewer defects.

Test Management

  1. Test Coverage - measuring the percentage of the application’s code covered by tests, manual & automated.
  2. Test Case Pass/Fail Ratio - measuring the percentage of test cases that pass and fail.
  3. Test Failure Rate - measuring the percentage of automated tests that fail.

In the fast-changing world of software development, QA metrics have an important role in tracking and sharing overall quality. They help teams make sure their products are good and keep getting better.

Using these metrics smartly turns QA from just testing into a way to manage quality strategically. This really helps software products do well in the market.

The QA role remains a crucial role of a software development team. As a frontrunner of quality, you as a software tester have the power to influence, innovate and inspire.