… Test Metrics Quantify Our Work
Simply put they allow Testers and Testing Managers to measure status at any given moment.
Usually these are standard queries or reports made in whichever ALM or Defect (Bug) Tracking Solution you have implemented.
Standalone Bug Tracking Systems can limit your effectiveness as there is a disconnect from Test Plans / Test Cases. How do you easily trace test coverage between your cases and your bugs?
There are solutions for this but let’s not digress…
A Cautionary Tale
I was in a meeting a while ago discussing preparations for certification compliance with Bob (Product Owner), Lucy (Compliance Officer) and Mary (Certification Manager).
Mary was asking questions about procedures, documentation, methodology and then she arrived at test metrics…
I keep queries and reports to show:-
Test Case coverage vs Specifications
Test Case Progress / version by module / tester
Bugs / version by severity by module / tester
Total bugs / version by severity by status…and a few others.
Mary was happy and then the other boot dropped. She recommended I use my Test Metrics to enable monitoring which testers weren’t working fast enough or reporting enough bugs and which developers were generating just too many bugs.
The next few moments of discussion are best left to your imagination.
After demonstrating onscreen the fact that not all test cases are created equal and prior knowledge is required to make sense of the scale of each module’s test cases in relation to others Mary was willing to listen to a more detailed explanation of how these test metrics should be used in a constructive, ethical manner.
The Warptest POV
Test Metrics are a useful tool but QA / Testing Managers need to apply caution when defining the metrics they measure, who has access to the metrics and to ensure a well thought out, structured explanation of the impact of the test metrics and their indication of overall status at any given stage of testing.
To paraphrase Churchill,
There are lies, damn lies and then there are testing metrics.
Make sure your metrics aren’t misleading you but more importantly make sure no one is abusing or being misled by the data you are feeding them.
Names were changed to protect the innocent and slightly guilty.
No ALM or Bug Trackers were harmed in writing this post.