Definition
Metrics are defined as "standards of measurement" and have long been used in the IT industry to indicate a method of gauging the effectiveness and efficiency of a particular activity within a project. Test metrics exist in a variety of forms. The question is not whether we should use them, but rather, which ones should we use. Simpler is almost always better. For example, it may be interesting to derive the Binomial Probability Mass Function for a particular project, although it may not be practical in terms of the resources and time required to capture and analyze the data. Furthermore, the resulting information may not be meaningful or useful to the current effort of process improvement.
One thing that makes Test Metrics unique, in a way, is that they are gathered during the test effort (towards the end of the SDLC), and can provide measurements of many different activities that have been performed throughout the project. Because of this attribute, Test Metrics can be used in conjunction with Root Cause Analysis to quantitatively track issues from points of occurrence throughout the development process. Finally, when Test Metrics data is accumulated, updated and reported on a consistent basis, it allows trend analysis to be performed on the information, which is useful in observing the effect of process changes over multiple projects.
Create Meaningful Metrics
Test Metrics are meaningful if they provide objective feedback to the Project Team regarding any of the development processes - from analysis, to coding, to testing - and support a project goal. If a metric does not support a project goal, then there is no reason to track it - it is meaningless to the organization. Tracking meaningless metrics wastes time and does little to improve the development process.
Metrics should also be objective. As indicated in the sample definition shown in the previous section, an objective metric can only be tracked one way, no matter who is doing the tracking. This prevents the data from being corrupted and makes it easier for the project team to trust the information and analysis resulting from the metrics. While it would be best if all metrics were objective, this may be an unrealistic expectation. The problem is that subjective metrics can be difficult to track and interpret on a consistent basis, and team members may not trust them. Without trust, objectivity, and solid reasoning, which is provided by the Test Metrics, it is difficult to implement process changes.
Track the Metrics
Tracking Test Metrics throughout the test effort is extremely important because it allows the Project Team to see developing trends, and provides a historical perspective at the end of the project. Tracking metrics requires effort, but that effort can be minimized through the simple automation of the Run Log (by using a spreadsheet or a database) or through customized reports from a test management or defect tracking system. This underscores the 'Keep It Simple' part of the philosophy, in that metrics should be simple to track, and simple to understand. The process of tracking test metrics should not create a burden on the Test Team or Test Lead; otherwise it is likely that the metrics will not be tracked and valuable information will be lost. Furthermore, by automating the process by which the metrics are tracked it is less likely that human error or bias can be introduced into the metrics.
Use Metrics to Manage the Project
Many times, members of a Project Team intuitively understand that changes in the development lifecycle could improve the quality and reduce the cost of a project, but they are unwilling to implement changes without objective proof of where the changes should occur. By meeting on a regular basis throughout a project, but especially at the end of a project, the team can review the Test Metrics and other available information to determine what improvements might be made. Here is an example of how metrics can be used to make process changes during a test effort.
Imagine that the Test team is halfway through the test execution phase of a project, and the Project Team is reviewing the existing metrics. One metric stands out - less than 50% of the test cases have been executed. The Project Manager is concerned that half of the testing time has elapsed, but less than half the tests are completed. Initially this looks bad for the test team, but the Test Lead points out that 30% of the test cases are 'Blocked'. The Test Lead explains that one failure in a particular module is preventing the Test Team from executing the 'Blocked' test cases. Moreover, the next test release is scheduled in 4 days and none of the 'Blocked' test cases can be executed until then. At this point, using objective information available from the metrics, the Project Manager is able to make a decision to push an interim release to the Test Team with a fix for the problem that is causing so many of the test cases to be 'Blocked'.
As seen by this example, metrics can provide the information necessary to make critical decisions that result in saving time and money and makes it easier for a project team to move from a "blame" culture to a "results-oriented" culture.
Types of Metrics
> Base Metrics
Base metrics constitute the raw data gathered by a Test Analyst throughout the testing effort. These metrics are used to provide project status reports to the Test Lead and Project Manager; they also feed into the formulas used to derive Calculated Metrics. Questcon suggests that every project should track the following Test Metrics:
| # of Test Cases |
# of First Run Failures |
| # of Test Cases Executed |
Total Executions |
| # of Test Cases Passed |
Total Passes |
| # of Test Cases Failed |
Total Failures |
| # of Test Cases Under Investigation |
Test Case Execution Time* |
| # of Test Cases Blocked |
Test Execution Time* |
| # of Test Cases Re-executed |
|
As seen in the 'Keep It Simple' section, many of the Base Metrics are simple counts that most Test Analysts already track in one form or another. While there are other Base Metrics that could be tracked, Questcon believes this list is sufficient for most Test Teams that are starting a Test Metrics program.
> Calculated Metrics
Calculated Metrics convert the Base Metrics data into more useful information. These types of metrics are generally the responsibility of the Test Lead and can be tracked at many different levels (by module, tester, or project). The following Calculated Metrics are recommended for implementation in all test efforts:
| % Complete |
% Defects Corrected |
| % Test Coverage |
% Rework |
| % Test Cases Passed |
% Test Effectiveness |
| % Test Cases Blocked |
% Test Efficiency |
| 1st Run Fail Rate |
Defect Discovery Rate |
| Overall Fail Rate |
Defect Removal Cost |
These metrics provide valuable information that, when used and interpreted, oftentimes leads to significant improvements in the overall SDLC. For example, the 1st Run Fail Rate, as defined in the Questcon Metrics Methodology, indicates how clean the code is when it is delivered to the Test Team. If this metric has a high value, it may be indicative of a lack of unit testing or code peer review during the coding phase. With this information, as well as any other relevant information available to the Project Team, the Project Team may decide to institute some preventative QA techniques that they believe will improve the process. Of course, in the next project, when the metric is observed it should be noted how it has trended to see if the process change was in fact an improvement.
The Final Step - Interpretation and Change
As mentioned earlier, test metrics should be reviewed and interpreted on a regular basis throughout the test effort and particularly after the application is released into production. During the review meetings, the Project Team should closely examine ALL available data, and use that information to determine the root cause of identified problems. It is important to look at several of the Base Metrics and Calculated Metrics in conjunction with one another, as this will allow the Project Team to have a clearer picture of what took place during the test effort.
If metrics have been gathered across several projects, then a comparison should be done between the results of the current project and the average or baseline results from the other projects. This makes trends across the projects easy to see, particularly when development process changes are being implemented. Always take note to determine if the current metrics are typical of software projects in your organization. If not, observe if the change is positive or negative, and then follow up by doing a root cause analysis to ascertain the reason for the change.
