There are a number of metrics that DevOps borrows from Lean, such as cycle time, lead time, WIP, MTTR and a few others. In order to determine how effective an organization is on their DevOps journey; the following metrics should be used to measure an organization’s DevOps effectiveness.
How often do you Deploy?
This metric is exactly what it says, how often do you deploy a new version of an application or service. The frequent deployment of applications or services suggests that the organization may have an effective continuous improvement practice in place. Frequent deployments also are a key in gaining business buy in.
This measures the amount of time that it takes to promote an application or service from one environment to the next. A proper deployment in DevOps assumes an application or service that is loosely coupled in order to be able to deploy without impacting the entire system.
Quality verification time
This is the amount of time that it takes for test automation to run and verify a new application or service. This time should continue on a downward slope if teams are focused on striving for green, meaning they are looking to continuously improve through automation.
Continuous Integration verification frequency
This is a key indicator of the organizations automated testing coverage progress. Without a high Continuous Integration verification frequency, the organization is most likely still handling builds and the corresponding testing in a manual manner. Having a manual build or testing verification, means the release pipeline process will never have a high level of confidence in the code moving through it.
Pipeline success rate
This metric tracks the success rate of iterative code releases through the delivery pipeline, this metric tells an organization if they have built high quality in the various steps in the release pipeline. This metric indicates if the organization has successfully dealt with “Drift” in the delivery pipeline and all of the components of automated testing, such as data management, test coverage and have they shifted left with testing (are they performing security, performance, consistency in the early stages of the pipeline).
Pipeline up time
This metric is an indicator of how often the delivery pipeline is up, which is an indicator of the organizations efforts towards automated testing, code quality, environment provisioning and Drift.
Specification by Example coverage ratio
This metric tracks the percentage of traceability between requirements and tests, which leads to higher confidence levels organization wide for code moving through the delivery pipeline.
Application/Service Feature usage
This metric is a key indicator of the DevOps feedback loops that the organization has put into place. Low usage signals a breakdown in communications in one of the channels between, customers, the business and IT.
Mean time to restore service (MTTRS)
This metric measures the average time it takes to restore a service or a function when the disruption is due to a defect that requires a fix to be developed. A high MTTR could indicate issues in the feedback loops, effectiveness of the delivery pipeline and the organizations ability to react to change.
Quality Gate Detection
This metric is a key indicator of the code quality being released through the delivery pipeline, which in turn results in lower technical debt.