Software Testing Metrics

 

Software Testing Metrics

Software Testing Metrics are the quantitative measures used to estimate the progress, quality, productivity and health of the software testing process. The goal of software testing metrics is to improve the efficiency and effectiveness in the software testing process and to help make better decisions for further testing process by providing reliable data about the testing process.

A Metric defines in quantitative terms the degree to which a system, system component, or process possesses a given attribute. The ideal example to understand metrics would be a weekly mileage of a car compared to its ideal mileage recommended by the manufacturer.

Software testing metrics - Improves the efficiency and effectiveness of a software testing process.

Software testing metrics or software test measurement is the quantitative indication of extent, capacity, dimension, amount or size of some attribute of a process or product.w to write a TEST CASE Software Testing Tutorial

 

Example for software test measurement: Total number of defects

In this tutorial, you will learn-

• What is Software Testing Metric?
• Why Test Metrics are Important?
• Types of Test Metrics
• Manual Test Metrics
• Test Metrics Life Cycle
• How to calculate Test Metric
• Example of Test Metric
• Test Metrics Glossary

Why Test Metrics are Important?

"We cannot improve what we cannot measure" and Test Metrics helps us to do exactly the same.

• Take decision for next phase of activities
• Evidence of the claim or prediction
• Understand the type of improvement required
• Take decision or process or technology change

Types of Test Metrics

• Process Metrics: It can be used to improve the process efficiency of the SDLC ( Software Development Life Cycle)
• Product Metrics: It deals with the quality of the software product
• Project Metrics: It can be used to measure the efficiency of a project team or any testing tools being used by the team members

Identification of correct testing metrics is very important. Few things need to be considered before identifying the test metrics

• Fix the target audience for the metric preparation
• Define the goal for metrics
• Introduce all the relevant metrics based on project needs
• Analyze the cost benefits aspect of each metrics and the project lifestyle phase in which it results in the maximum output

Manual Test Metrics

In Software Engineering, Manual test metrics are classified into two classes

• Base Metrics
• Calculated Metrics

Base metrics is the raw data collected by Test Analyst during the test case development and execution (# of test cases executed, # of test cases). While calculated metrics are derived from the data collected in base metrics. Calculated metrics is usually followed by the test manager for test reporting purpose (% Complete, % Test Coverage).

Depending on the project or business model some of the important metrics are

• Test case execution productivity metrics
• Test case preparation productivity metrics
• Defect metrics
• Defects by priority
• Defects by severity
• Defect slippage ratio

Example of Test Metric

To understand how to calculate the test metrics, we will see an example of a percentage test case executed.

To obtain the execution status of the test cases in percentage, we use the formula.

Percentage test cases executed= (No of test cases executed/ Total no of test cases written) X 100

Likewise, you can calculate for other parameters like test cases not executed, test cases passed, test cases failed, test cases blocked, etc.

Test Metrics Glossary

• Rework Effort Ratio = (Actual rework efforts spent in that phase/ total actual efforts spent in that phase) X 100
• Requirement Creep = ( Total number of requirements added/No of initial requirements)X100
• Schedule Variance = ( Actual efforts – estimated efforts ) / Estimated Efforts) X 100
• Cost of finding a defect in testing = ( Total effort spent on testing/ defects found in testing)
• Schedule slippage = (Actual end date – Estimated end date) / (Planned End Date – Planned Start Date) X 100
• Passed Test Cases Percentage = (Number of Passed Tests/Total number of tests executed) X 100
• Failed Test Cases Percentage = (Number of Failed Tests/Total number of tests executed) X 100
• Blocked Test Cases Percentage = (Number of Blocked Tests/Total number of tests executed) X 100
• Fixed Defects Percentage = (Defects Fixed/Defects Reported) X 100
• Accepted Defects Percentage = (Defects Accepted as Valid by Dev Team /Total Defects Reported) X 100
• Defects Deferred Percentage = (Defects deferred for future releases /Total Defects Reported) X 100
• Critical Defects Percentage = (Critical Defects / Total Defects Reported) X 100
• Average time for a development team to repair defects = (Total time taken for bugfixes/Number of bugs)
• Number of tests run per time period = Number of tests run/Total time
• Test design efficiency = Number of tests designed /Total time
• Test review efficiency = Number of tests reviewed /Total time
• Bug find rote or Number of defects per test hour = Total number of defects/Total number of test hours

Explains in detail with example about calculation of -
1.Percentage Test cases Executed [Test Coverage]
2.Percentage Test cases not executed
3.Percentage Test cases Passed
4.Percentage Test cases Failed
5.Percentage Test cases BLOCKED/Deferred
6.Defect Density
7.Defect Removal Efficiency (DRE)
8.Defect Leakage
9.Defect Rejection ratio [Invalid bug ratio]
10.Percentage of Critical defects
11.Percentage of High defects
12.Percentage of Medium defects
13.Percentage of Low/Lowest defects

 Software Test Metrics and Measurements

1. 1. Test Metrics “We can’t control things which we can’t measure” Presented by : DavisThomas K
2. 2. What are Software Testing Metrics? – A Metric is a quantitative measure of the degree to which a system, system component, or process possesses a given attribute. – Software Metrics are used to measure the quality of the project.
3. 3. Test Metrics are used to – Evaluate testing performance and level of quality – Based on metrics , a better trade off between constraints is possible – Understand the kind of improvement required to success the project – Take decision on process or technology to be modified – Test Metrics are the most important to measure the quality of the software.
4. 4. Metrics Life cycle
5. 5. Classification Test Metrics Base [Direct] Metrics Calculated [Indirect] Metrics Base metrics is the raw data collected by Test Analyst during the test case development and execution Calculated metrics is derived from the data gathered in base metrics.
6. 6. Basic test Metric
7. 7. Test Report dashboard
8. 8. Estimating testing BUG REPORT
9. 9. Calcu lated Metri cs conv ert the Base Metri cs data into more Percentage Test cases Executed Percentage Test cases not executed Percentage Test cases Passed Percentage Test cases Failed Percentage Test cases Blocked Defect Density Defect Removal Efficiency (DRE) Defect Leakage Defect Rejection ratio Defects by Priority Calculated test metrics Calculated Metrics convert the Base Metrics data into more useful information.
10. 10. PercentageTest cases Executed [Test Coverage] – To obtain percentage of test case executed from total test case – Test Coverage= (No. ofTest cases executed /Total no. ofTest cases written) * 100. Test Coverage = (600/ 640) * 100 = 93.75%
11. 11. PercentageTest cases NOT Executed To obtain percentage of test case NOT executed from total test case – Test Coverage= (No. ofTest cases NOT executed /Total no. ofTest cases written) * 100. Test Coverage = (40/ 640) * 100 = 6.25%
12. 12. Percentage ofTest cases passed To obtain percentage of test case passed – PercentageTest cases Passed = (No. ofTest cases Passed /Total no. ofTest cases Executed) * 100. Percentage of test cases Passed = (510/ 600) * 100 = 85%
13. 13. Percentage ofTest cases failed To obtain percentage of test case failed – PercentageTest cases Failed= (No. ofTest cases failed/ Total no. ofTest cases Executed) * 100. Percentage of test cases Failed= (80/ 600) * 100 = 13.33%
14. 14. Percentage ofTest cases blocked /deferred This metric is used to obtain the blocked %ge of the executed test cases.A detailed report can be submitted by specifying the actual reason of blocking the test cases. – PercentageTest cases blocked= (No. ofTest cases blocked/Total no. ofTest cases Executed) * 100. Percentage of test cases blocked= (10/ 600) * 100 = 1.67%
15. 15. Defect Density Defect density could be defined as the value of the total defects which are known to the size of the software product calculated Defect Density = (32/ 50) = 1.67 defects / Story Here, Size of release can be measured in terms of line of code (LoC), user story, Functional point.
16. 16. Advantages of defect density •It helps measure the testing effectiveness •It helps to differentiate defects in components/software modules •It is useful in identifying the areas for correction or improvement •It is useful in pointing towards high-risk components •It helps in identifying the training needs to various resources •It can be helpful in estimating the testing and rework due to bugs •It can estimate the remaining defects in the software •Before the release we can determine whether our testing is sufficient
17. 17. Defect Removal Efficiency (DRE) DRE is used to identify the test effectiveness of the system. The defect removal efficiency (DRE) gives a measure of the development team ability to remove defects prior to release. It is calculated as a ratio of defects resolved to total number of defects found. Defect Removal Efficiency (DRE) = (No. of Defects found during QA testing / (No. of Defects found during QA testing +No. of Defects found by End user)) * 100 DRE = [29/ (29+ 3)] * 100 = [29/32] * 100 = 90.63%
18. 18. Defect leakage – Defect Leakage is the Metric which is used to identify the efficiency of the QA testing i.e., how many defects are missed / slipped during the QA testing. – Defect Leakage = (No. of Defects found in UAT / No. of Defects found in QA testing.) * 100 DRE = [3/ 29] * 100 = 10.34%
19. 19. Defects by Priority Percentage of Critical defects – Percentage of Critical Defects = No. of Critical Defects identified /Total no. of Defects identified * 100 Percentage of Critical Defects = (10/ 32) * 100 = 31.25%
20. 20. Defects by Priority Percentage of High priority defects – Percentage of High priority Defects = No. of High priority Defects identified /Total no. of Defects identified * 100 Percentage of Critical Defects = (5/ 32) * 100 = 15.63%
21. 21. Defects by Priority Percentage of Medium priority defects – Percentage of Medium priority Defects = No. of Medium priority Defects identified /Total no. of Defects identified * 100 Percentage of Medium priority Defects = (12/ 32) * 100 = 37.5%
22. 22. Defects by Priority Percentage of Low priority defects – Percentage of Low priority Defects = No. of Low priority Defects identified /Total no. of Defects identified * 100 Percentage of Low Priority Defects = (3/ 32) * 100 = 9.38%
23. 23. Defect Rejection ratio [Invalid bug ratio] – Percentage of Invalid Defects = No. of Invalid Defects identified /Total no. of Defects identified * 100 Percentage of Invalid Defects = (2/ 32) * 100 = 6.25%