Determining the success of a product development and test effort is a matter of ensuring that the organization has delivered a product that meets its customers’ quality expectations. An organization must measure several dimensions of the development and testing to determine whether it has been successful in meeting quality goals.
Developing a measurement strategy
A measurement strategy should address four key aspects:
•Process: Metrics for the effectiveness and efficiency of the test process, which address the consistency of test efforts, either by comparing current efforts with past performance or by measuring the consistency of test activities for different teams within an organization.
•Quality: Metrics that address the quality of the entity being tested, typically it addresses both the volume and the severity of defects.
•Schedule: Metrics that assess the overall schedule with adherence to the test plan or comparing the duration per test phase against past programs.
•Cost: Metrics that measure investments in a particular test project.
Building a measurement infrastructure
Prior to implementing a measurement strategy, an organization must create a measurement infrastructure to track the number and type of defects reported during the development and test process as well as the number of resources allocated to test activities.
Defect-tracking system
A defect-tracking system can be as simple as a local database or as sophisticated as one of the many defect-tracking tools available today. A key characteristic of a defect-tracking system is the ability to categorize defects by problem area, state, and severity. The state of a defect is its current phase in the defect life cycle. Defect phases include created, fixed, closed, assigned, and rejected.
Resource-tracking system
A resource-tracking system is necessary to measure the number of hours spent on test activities. Useful metrics include the number of total hours spent on the project (for planning, execution, and reporting), the number of manual or automated test hours, and the number of hours spent during each test phase.
Goals and standards
Goals and standards are the benchmarks in a measurement system. These may be defined as a percentage improvement over historical programs, or they may be based on an industry source or a best practice from another organization doing similar work.
Identifying key measurements
A measurement system needs to provide metrics in three categories: operational, project assessment, and business.
Operational metrics
Operational metrics are designed to measure the project progress during the test execution phase—whether that phase is unit, product, or system testing. Organizations may use these metrics to assess the stability of the product at a given point of time, release readiness, adherence to test entrance and exit criteria. Common operational metrics are: Defect management, Defect arrival rate, Defect incoming/ closure gap, Defect aging, and Test execution progress.
Project assessment metrics
A key objective of any organization is to execute the test cycle for a given project according to the test plan. Project assessment metrics are designed to provide the organization with feedback on the efficiency of test planning, execution, and reporting efforts. Common project assessment metrics are: Mean time to close, Test plan effectiveness, Invalid defect rate, Adherence to resource plan, Defect incoming rate by test phase, Defect closure rate by test phase, Volatility index, and Schedule adherence.
Business metrics
Business metrics address whether an organization is executing tests with consistency and improving in its test efforts. It compares test planning, execution, and reporting efforts with historical efforts of a similar type. Business metrics are typically reviewed monthly or quarterly. Common business metrics are: Defect discovery accuracy, Test plan effectiveness, Test case execution profile, Adherence to resource plan, Adherence to capital plan, Field escapes, Defect incoming rate by test phase, Defect closure rate by test phase, and Schedule adherence.
Gauging the success of test efforts
The primary goal of measuring the development and test process is to ensure that enterprises are meeting the quality expectations of customers. Quality measures indicate whether a product is performing to the specifications defined by the marketing team and implemented by the product development group.