In today’s IT world a frequent term that is being thrown around in companies, conferences, meetups, blogs and articles is the buzzword “DevOps” Unfortunately these discussions regarding DevOps often fail to understand the background behind it, what it actually encompasses or try to use it as a false selling point.
In order to understand DevOps there are a couple of key points that must be understood in order to fully grasp what it entails. DevOps should be looked at as a natural progression of earlier technology and process trends that have swept the IT industry. Looking historically we see practices like ITIL that have helped IT operations increase stability and reliability, the Agile methodology which has speeded up the flow and quality of work coming out of development, processes such as Continuous integration/deployment that have enabled frequent builds and deployments and the growth of cloud services which has allowed for the delivery of mammoth services.
In order to truly comprehend what DevOps is, where it came from, its implications and history we need to take a close look at the “Lean” manufacturing revolution and the teachings of the “big three” and others during the manufacturing quality revolution. A major emphasis during the quality revolution was a quality-focused approach to improving performance that stressed the use of statistical methods. While many individuals may be unfamiliar with the Lean manufacturing revolution, the quality changes it brought about or the leaders of this revolution. Most individuals are familiar with the company that is regarded as the pinnacle of this revolution, the Toyota motor company. Many processes, terms and concepts came out of Toyota and other companies during this time, such as Kanban, Kaizen, Just in Time (JIT), Work in Progress (WIP), Quality Circles (Feedback Loops) and the Theory of Constraints.
When we look back at the Lean manufacturing revolution we see many similarities to what DevOps faces today, the belief by many that changing the culture and the practices in large, complex organizations is difficult and thus is unlikely.
The reality is that the Lean manufacturing revolution wasn’t easy either. It required changing incentives for plant managers, work center supervisors and even business managers. It required overcoming work center silos, massive levels of training and re-skilling, and an integrated workforce focused on flow, as opposed to units produced.
Many organizations worldwide refused to embark on this journey, and others started but did not have skill or collective will to solve this problem. Those are the companies that are no longer relevant.
Many companies and individuals have not fully grasped that the DevOps transformation is well underway, not just in startups, but in enterprises in all industry verticals and sizes, from retailing, to financial services, health care and even government agencies and non-profits.
Companies who fail to grasp the magnitude of the DevOps revolution risk missing out on one of the most disruptive and innovative periods in technology history.
Many of the original DevOps leaders believe that the DevOps revolution has the ability to be even larger than what was created by the manufacturing revolution. The reasoning behind this is due to IT being the factory floor of this century. It has become increasingly clear that IT is how businesses acquire customers and deliver value to them.
By fully adopting DevOps and transforming the IT value stream companies are likely to see a productivity surge as large, or larger, than the Lean manufacturing revolution, making this one of the most important business opportunities in IT history.
Many companies and individuals when discussing DevOps will often lament that they have already tried the Agile methodology and implemented CI and CD tools, but have not seen any of the results expounded by the DevOps followers. The truth is that these companies and individuals have typically grabbed the low hanging fruit when implementing Agile and implementing CI/CD tools and have failed to tackle the tough changes required to be fully Agile and correctly do CI/CD and move towards DevOps as shown below.
When looking at the Waterfall methodology a critical problem standouts when looking at the model above. The next image shows an underlying problem with the Waterfall methodology.
When we look at Waterfall we see clear and distinct boundaries between departments that have their own silos. Inside these silos we see departments that own their own tools and processes. The creation of silos results in a lack of collaboration, nonstandard tools and disjointed processes. Agile has attempted to bridge these boundaries by breaking down some of the silos as shown below.
If done correctly, Agile can be very successful in breaking down these silos and making organizations more productive and nimble, but the truth is that very few organizations have done Agile correctly. If we are honest, the image below shows what has actually happened at the majority of organizations who have adopted Agile.
The truth is that most organizations who have adopted the Agile methodology have quote “adopted agile practices”, in truth they have cherry picked the pieces that are fairly easy to implement and ignored the difficult changes. The reality is that the agile teams that have been created are still encased in their silos. If you look closely at the agile teams, they still have the same reporting structure in place, use their own tools and rely on their traditional processes. Another issue also rears its head as the Agile process moves forward, the new code being developed by these teams is not getting released to production any quicker. This has unfortunately led to many parts of the organization and management to look at Operations as a bottleneck, which in turn has led to the attempts below to fix the bottleneck as shown below.
A couple of common fixes that agile practitioners and organizations use to attempt to fix the perceived bottleneck at Operations is to either include an Operations member in the Agile teams or implement Kanban in Operations. These attempts are doomed to fail because they ignore the underlying issues such as Operations and everyone on the Development side having misaligned incentives as shown below. In looking at the bottleneck we need to look at the “Theory of Constraints” introduced by Eliyahu Goldratt who preached the importance of cycle time reduction, increasing throughput, and treating manufacturing as a system whose performance is controlled by the bottleneck process
When observing the technology companies who have been able to successfully navigate the road to DevOps we find they have many commonalities. These organizations actively avoid silos and encourage collaboration across boundaries. Most importantly, they are able to sustain a level of performance that hardly seems possible to the rest of the world.
The Building Blocks of DevOps
• Blame Game
• Conways Law
• Evangelism/Public relations
• Overcoming organizational barriers
• Moving from a “ticket” culture to “ownership” culture
• Cross-functional teams with end-to-end ownership of services
• Overcoming traditional project attributes
o Hero cult
o Shadow responsibilities
o Emphasis on title
o Favor a plan over planning
• Mike Rother in his “Toyota Kata” book points out that of the many who try to emulate Toyota, most miss the invisible side of what they were doing. The key, as Rother suggests, is that it wasn’t the tools that made Toyota great, it was the culture and specific behavior associated behind those tools.
Measurement and Metrics
• “Change” serves as a shared unit of measurement.
• Cycle Time
• Focus on leading quality metrics instead of lagging
• Definition of Done
• SonarQube (Required to be part of CI, quality gates)
• Measure DevOps Progress (Specification by Example)
• Measure Cultural Change (surveys)
Improve and Accelerate Delivery
• Incremental releases
• SCM Required
• By reducing batch size, we can deploy more frequently because reducing batch size drives down cycle time.
• The key goal of automatic releasing is to reduce the risk of releasing.
• Should you automate everything? NO
• Law of Marginal Costs
• Required to flip the testing pyramid
Apply Releases Incrementally and Iteratively
• An iteration is a mini-project that may result in an increment of the software. Iterating starts with an idea of what is wanted, and the code is refined to get the desired result. An increment is a small unit of functionality. Incrementing allows you to build a better understanding of what you need, assembling the software piece by piece.
• Upgrading all of the components in one big-bang release is the highest-risk way of rolling out new functionality.
• Instead, deploy the components independently in a side-by side configuration wherever possible.
• Automatic releasing must be accompanied by monitoring.
• Monitoring is the activity of continuously collecting and storing data about the state of the application, middleware, and infrastructure and making this state visible to the whole team.
• Monitoring is used to detect (or even prevent) production incidents and to minimize MTTR and MTTD. (Mean Time to Resolution/Detection)
• Monitoring Driven Development – The Holy Grail – Health Dashboard
• Running Automated Tests in Production – Yes
• Smoke Tests – There is a reason we do this
• DevOps fundamental – Never break the customer (internal or external)
Decoupled Deployment and Release
• Decoupling deployment and release improves and accelerates delivery, which is one building block of DevOps.
• We no longer use traditional branching strategies.
• Branch by Abstraction
• Feature Toggles
• Dark Launching
• Blue-Green Deployment
Specification by Example
• Behavior Driven Development/Acceptance Driven Development (BDD/ATDD)
• DevOps requires a common language (Release example)
• Living/executable documentation
• It provides the traceability for everything you’re doing
Infrastructure as Code
• We treat the infrastructure the exact same as application code.
• Lives in an SCM
• Is tested just like application code (TDD)
• Is monitored constantly just like application code
Additional DevOps Benefits
• A/B Testing
• Performance and Load Testing
• Chaos Monkey
• Mobile (treat just like binary files)
• Moving away from outsourcing and investing in engineering practices
Cutting Edge DevOps Practices
• Information Radiators
• Internal DevOps Days
• Internal Software Engineering Conferences
• DevOps/IT Academies
• Engineering Blogs
• Communities of Practice
• Jidoka/Andon Cord
• Chaos Cord
• Improvement Kata
• “Sensei”, or Masters