How to Measure DevOps Implementation

In my last article, we discussed on how Agile, DevOps and DevSecOps are related. Start implementing all these approaches and practices are quite straight forward, but then how to know that we are in right direction? This is when the measurement plays a big role.

Why Measurement?

“If you cannot measure it, you cannot improve it”, this famous quote from Peter Drucker stresses on the importance of measurement based on actual data to know our current condition and find the opportunities to continuously improve. 

If your Enterprise is implementing DevOps with all the automation, most of the teams will leverage CI/CD pipeline to deliver ready-to-deploy and release package with quality, and ALL of the processes in this pipeline should be measured and monitored.

Some metrics such as DORA Metrics and Flow metrics are the most general metrics to provide objective data to compare the performance of delivery and to drive improvement.

The DORA Metrics 

The DevOps Research and Assessment (DORA) metrics are commonly used by Agile Product / Business Team (and also by platform engineering or release engineering) to measure their delivery performance and stability.

The four DORA metrics :plus: one additional reliability metric (I believe this also important) are as below.

Deployment FrequencyRefers to the frequency of successful software releases to production in a certain period of time.
Lead Time for Changes / Deployment Lead TimeCaptures the time between a code change commit and deployment to target environments, usually until to production environment.
Time to Recover (or MTTR)Measures the time between an interruption due to deployment or system failure and full recovery.
Change Failure RateIndicates how often a team’s changes or hotfixes lead to failures after the code has been deployed. It’s the percentage of deployments that trigger one or more production failures, for all deployments of a service per measurement interval in a measurement period.
Reliability (Additional)Measure the service level objective (SLO) that we agree to meet with business expectation. I will discuss SLO in my next article, so stay tuned.

Following is the data from Google’s 2021 DevOps State of Report for DORA metrics, and can be used as comparison to your team’s delivery.

DevOps Metrics Table Comparison

By retrieving data from CI/CD pipeline, your team can visualize and measure using these metrics, and find the opportunity for improvement.

Team can also observe the trend to identify factors that improve / decrease the performance, and agree on the improvement points.

Example in graph below showing on May or June, the interval time for deployment is increasing, means some conditions might be happened that slow down our delivery, for example : test environment creation, test data, or anything else.

Performance Trend

By measuring the actual delivery process, we can find many improvement opportunities. You can also introduce some additional and meaningful measurements, depends on your objective (e.g defect density, security rating/quality, flow efficiency, number of #manual steps, etc.)

Always remember that the end goal is not to implement Agile and Dev(Sec)Ops, but on how to deliver value better and securely with highest quality.

Always introduce small improvement, make it part of your daily process and culture. The goal is not to be perfect, but to be as close as we can be.

Stay nimble and agile with Agile DevSecOps!