- TL;DR
- What are the testing metrics?
- QA Metrics
- Product (quality) risks, Defects, Tests, Coverage, Confidence
- Main metrics: What is planned? What is tested? No. of test cases, no. of test cases ready for automation, execution reports. Coverage report.
- Automation metrics
TL;DR
- Test coverage (what is planned, what we have achieved now, how many smoke/regression)
- Automation coverage (how many ready_for_automation, number/percentage of automated test cases)
- Defects metrics (found, fixed, how many found by automation)
- Test execution results: Run, Passed, Failed, Blocked, Skipped. Time spend for executing. Metrics by modules.
- Testing frequency (what is planned, what is actual)
- Test efficiency (defects found by you and defects found by customer)
- Sprint metrics (what did we do during this sprint)
What are the testing metrics?
Metrics help us measure the effectiveness of testing, test cases, if we have bugs → that means that metrics are not effective enough. Estimate progress
- Metrics measure progress, quality, productivity of the testing, and quality of product.
- Help make better decisions for further testing.
Why test metrics are important?
"We cannot improve what we cannot measure" and Test Metrics helps us to do exactly the same.
- Take decision for next phase of activities
QA Metrics
- Project metrics: measure progress toward established project exit criteria, such as the percentage of test cases executed, passed, and failed
- Product metrics: measure some attribute of the product, such as the extent to which it has been tested or the defect density
- Process metrics: measure the capability of the testing or development process, such as the percentage of defects detected by testing
- ВСЕ МЕТРИКИ ДОЛЖНЫ БЫТЬ СОГЛАСОВАНЫ ДО ИХ ПРИМЕНЕНИЯ
- ОБЯЗАТЕЛЬНОЕ ОТСЛЕЖИВАНИЕ В ДИНАМИКЕ (automatically), ВСЕ МЕТРИКИ ПРОСЧИТЫВАЮТСЯ КАЖДУЮ ИТЕРАЦИЮ И АНАЛИЗИРУЮТСЯ
- ОСНОВНЫЕ ИЗ НИХ ДОЛЖНЫ ПОПАДАТЬ В ОТЧЕТ ПО РЕЗУЛЬТАТАМ ТЕСТИРОВАНИЯ И ПРЕЗЕНТОВАТЬСЯ КОМАНДЕ
Product (quality) risks, Defects, Tests, Coverage, Confidence
Product (quality) risks
- % of risks completely covered by passing tests
- % of risks for which some or all tests fail
- % of risk not yet completely tested
- % of risks covered, sorted by risk category
- % of risks identified after the initial quality risk analysis
Defects
- Reported / Fixed
- Time between failure or failure arrival rate
- Breakdown of the number or percentage of defects categorized by the following:
- Particular test items or components
- Root causes
- Source of defect (e.g., requirement specification, new feature, regression, etc.)
- Test releases
- Phase introduced, detected, and removed
- Priority/severity
- Reports rejected or duplicated
- Trends in the lag time from defect reporting to resolution
- Number of defect fixes that introduced new defects (sometimes called daughter bugs)
Tests
- Total number of tests planned, specified (implemented), Run, Passed, Failed, Blocked, Skipped
- Regression and confirmation test status, including trends and totals for regression test and confirmation test failures
- Hours of testing planned per day versus actual hours achieved:
- Availability of the test environment (percentage of planned test hours when the test environment is usable by the test team)
Test coverage
- Requirements and design elements coverage
- by test-cases
- by auto (current and planned value)
- by execution for a phase
- Risk coverage
- Environment/configuration coverage
- Code coverage
Main metrics: What is planned? What is tested? No. of test cases, no. of test cases ready for automation, execution reports. Coverage report.
- Requirements traceability matrix = document that maps and traces user requirement with test cases.
Дефекты
- Общее количество найденных багов
- Общее количество неисправленных багов на проде
- Баги в разрезе приоритета/серьезности
- Баги в разрезе модуля. На основании этого можно посчитать плотность дефектов в отдельном модуле: баги в модуле/общее количество дефектов
- Непотвержденные дефекты: число дефектов в статусе FAD/общее количество дефектов
- Повторно открытые дефекты: повторно открытые баги/общее число дефектов
Test Кейсы
- Общее количество написанных кейсов
- Общее количество автотестов
- Процент автоматизированных смоук-кейсов от общего числа
- Процент автоматизированных регрессионных кейсов от общего числа
- Процент автоматизированных кейсов с высоким приоритетом от общего числа
- Процент покрытия требований манульными тестами: общее число кейсов/общее число требований
Оценка трудозатрат
- Общее количество времени на регрессионное тестирование
- Количество времени, затраченное на регрессионное тестирование с учетом автоматизации
- Количество времени затраченного на смоук
- Количество времени, затраченное на смоук тестирование с учетом автоматизации
- Точность эстимации = оценочное время/фактическое время. Полученный коээфициент можно учитывать для будущих эстимаций
Дополнительные метрики
- Оценка качества кода разработчика = дефекты в коде разработчика/общее количество дефектов
- Удовлетворенность пользователей: опросы, оценки в маркетах
Automation metrics
- Number/Percent of test cases that are ready for automation
- Automation progress, how many tests are automated already
- Regression frequency - how frequent do we run tests
