DevOps Handbook Summary 2 of 4 - The Second Way

Book summary of The DevOps Handbook by Gene Kim et. al. Excerpted content is formatted in italics.

Part IV: The Second Way - The Technical Practices of Feedback

Part four explores:
  • Creating telemetry to enable seeing and solving problems 
  • Using our telemetry to better anticipate problems and achieve goals 
  • Integrating user research and feedback into the work of product teams 
  • Enabling feedback so Dev and Ops can safely perform deployments 
  • Enabling feedback to increase the quality of our work through peer reviews and pair programming

Create telemetry to enable seeing and solving problems

  • Strong DevOps performers have Mean Time To Recover (MTTR) of 165x better than their peers w/80% of the MTTR time typically spent determining what changed (see Puppet Labs annual DevOps reports
  • Two top predictors of DevOps success:
    • Operations (and Infrastructure) code is highly automated and under source control
    • Proactive monitoring and telemetry
The authors share Turnball's a best practice framework for monitoring (telemetry collection and consumption):

Key characteristics:
  1. Covers all layers: business, application, infrastructure, client software, deployment pipeline
  2. Centralize data for ease of access
  3. Self-service enabled for creation, access, analysis etc.
  4. Highly available
Make it easy to on a daily basis:
  • Create telemetry
  • Search
  • Relate telemetry to 
    • production changes, deployments, release events
    • business events etc.
  • See - provide information radiators to all major stakeholders: Business (e.g. A/B test outcomes, usage anomalies...), Operations and Infrastructure, Development, Security...
Choosing the right logging level is important. Dan North, a former ThoughtWorks consultant who was involved in several projects in which the core continuous delivery concepts took shape, observes, “When deciding whether a message should be ERROR or WARN, imagine being woken up at 4 a.m. Low printer toner is not an ERROR.” 

To help ensure that we have information relevant to the reliable and secure operations of our service, we should ensure that all potentially significant application events generate logging entries, including those provided on this list assembled by Anton A. Chuvakin, a research VP at Gartner’s GTP Security and Risk Management group:
  • Authentication/ authorization decisions (including logoff) 
  • System and data access 
  • System and application changes (especially privileged changes) 
  • Data changes, such as adding, editing, or deleting data Invalid input (possible malicious injection, threats, etc.) 
  • Resources (RAM, disk, CPU, bandwidth, or any other resource that has hard or soft limits) 
  • Health and availability Startups and shutdowns 
  • Faults and errors 
  • Circuit breaker trips 
  • Delays 
  • Backup success/ failure
To make it easier to interpret and give meaning to all these log entries, we should (ideally) create logging hierarchical categories, such as for non-functional attributes (e.g., performance, security) and for attributes related to features (e.g., search, ranking).

At the heart of monitoring and telemetry is the need for healthy organizations to make the reality of complex environments visible. We must avoid the counter productive, blame based "mean time until proven innocent" culture that rumor and hearsay that replaces the lack of readily available facts from across the platform (app, ops, infra, sec) to consider.

Analyze telemetry to better anticipate problems and achieve goals

Goal: Create tools to find ever weaker failure signals in production telemetry so we can avoid catastrophic failures

Key idea: Identify outliers in gaussian and non-gaussian data sets i.e. having normal and other distributions  e.g. chi-squared ops data

Netflix used outlier detection to identify servers among thousands that were unusual and likely misbehaving. They didn't try to understand what was wrong. It was easier and faster to just destroy and recreate the servers. This improved the customer experience and cost much less than trying to analyze the outliers problems.

As usual the overall process (the what) is relatively simple and the implementation (the how) is tricky. In short:
  1. Analyze the data collected during severe incidents
  2. Identify the telemetry that would have enabled faster detection and diagnosis and alert on them
  3. Identify the telemetry that would have verified that the problem was resolved
  4. Continue this process to identify ever weaker failure signals 
Many operational data sets are not gaussian ie. not bell curved, Apply smoothing techniques to avoid over alerting and creating operational noise. Examples include: fast furrier transformation, linear regression, rolling averages etc.

A handful of example charts from the book to illustrate:

Feedback enables development and operations safely deploy code and release features

DevOps success = automated deployments + integrated monitoring and telemetry + shared feeling of responsibility for the health of the entire value stream.

  • Small, frequent changes that all can inspect and understand i.e. smooth, continuous flow
  • Drive out fear! Focus on Mean Time To Recover rather than Mean Time Between Failures
  • Tie telemetry reporting to deployments as they are typically, the top cause for failure
How do we inspire the needed collaboration and shared accountability across development, infrastructure, operations etc?  The authors aim to create the conditions of empowered empathy by enriching the experiences of their people:
  • Have developers experience the up and downstream impacts of their choices and omissions
    • Pain users feel
    • Pain downstream players feel keeping brittle things running
  • Have developers work with operations, infrastructure etc. identify telemetry and automation opportunities 
Organizational practices:
  • Developers self-manage their applications post release for some amount of time prior to operations acceptance. 
  • If you don't have it consider creating Launch Guidance. Launch guidance and requirements will likely include the following: 
    • Defect counts and severity: Does the application actually perform as designed? 
    • Type/ frequency of pager alerts: Is the application generating an unsupportable number of alerts in production? 
    • Monitoring coverage: Is the coverage of monitoring sufficient to restore service when things go wrong? 
    • System architecture: Is the service loosely-coupled enough to support a high rate of changes and deployments in production? 
    • Deployment process: Is there a predictable, deterministic, and sufficiently automated process to deploy code into production? 
    • Production hygiene: Is there evidence of enough good production habits that would allow production support to be managed by anyone else?
  • Design for operations i.e. include definition of telemetry, monitoring and other operational requirements at the outset of every IT project
  • For functional organizations consider Google's Service Reliability Team:

Hypothesis driven development and A/B testing

Fast, safe deployments enable us to learn from our customers actual vs. predicted behavior based on product changes. The Lean Start Up deep dives this much broader topic e.g. the value hypothesis and growth hypothesis.

A/B or Split Test website example:
  • Create two cohorts - control and treatment
  • Assign the current feature to the current state
  • Assign new feature to the treatment 
  • Statistically significant results determine success or failure of a new feature
Some great feature ideas actually make things worse - despite our organizations belief to the contrary.

Typical hypothesis example:
  • We believe bigger pictures of hotels on the booking page
  • Will result in improved conversion rates
  • We will adopt if we see at least a x% conversion improvement

Create review and coordination processes to increase the quality of our current work

This chapter of the book focusses on an irony that the authors each find all too familiar. High profile incidents tend to generate two "counter factual arguments" i.e. arguments fulfilling our human tendency to quickly identify possible alternatives (that seem right) rather than being right or factually correct. The two most common being:
  1. This was a change control failure - we should've had better reviews in place
  2. This was a testing failure - they should have found this sooner and recovered from it faster  
These arguments typically lead to a downward spiral: add more reviews from often external reviewers. These reviewers are farther away from the context.  These reviews require more complex coordination and lead to reviewing larger batch sizes.  The distance and batch size reduce the effectiveness of the review process and ironically, tend to reduce rather than improve quality leading to more severe and frequent production issues rather than less. One of the authors began their careers as an auditor and found this irony a very painful lesson to learn.

The authors basically argue, based on value stream analysis, that the only practical means of ensuring quality is to build it in via:
  • Small batches of
  • Loosely coupled units of code
  • Designed to address relevant functional and non-functional requirements
  • Continuously integrated and tested via automated tests and
  • Frequently reviewed by competent staff closest to the work
And they point out that the Agile practices of eXtreme Programing enable all of the above.

Other key practices: 
  • Pull requests (peer review)
  • Static code analysis tools
  • etc.
These practice are deceptively simple i.e. they are hard to get right. They depend on people, bottom up to self-organize to address the inherent human and procedural complexities of software delivery and operation. 

Supporting charts from the book:

External, cross organizational boundary reviews of high risk changes are still needed an can add significant value.  But must be carefully managed and appropriately limited because building quality into the product as shown above presents exponentially higher returns and lower risks.

Next up: DevOps Handbook Summary 3 of 4 - The Third Way


  1. its a great article on your blog about devops. thank u for sharing
    Devops Online Training india

  2. thank you for providing the valuable information regarding devops course information ..keep update with your blogs...once check it out
    Devops Online Training Hyderabad

  3. Great put Good stuff.All the topics were explained quickly understand for me.I am waiting for your next fantastic blog.Thanks for sharing.Any coures related details learn...
    Devops Online Training

  4. Thank you so much for sharing this worth able content with us. The concept taken here will be useful for my future programs and i will surely implement them in my study. Keep blogging article like this.

    Devops Training in Marathahalli|
    Devops Training in Bangalore|

  5. This blog is really awesome Thanks for sharing most valuable information with us. check at Devops Online Training Hyderabad

  6. Nice and good article. It is very useful for me to learn and understand easily.
    DevOps Online Training

  7. If you want to Learn salesforce CRM Course Contact Navya from onlineitguru. onlineitguru Is one of the Leading Training institute to Offer salesforce Online Training Bangalore. For more Details Call@8885448788.

  8. Nice blog. Explained well. I have suggested to my friends to go through this blog. Very nice explanation. Thank you for sharing this useful information.If you want to learn devops course in online, please visit below site.
    devops Online Training, devops course, devops online training in kurnool, devops online training in hyderabad, devops online training in bangalore, online training, trending courses

  9. Great Site, The Course were so simple and easy to understand.
    DevOps Online Training

  10. Thanks for the well-written post and I will follow your updates regularly and this is really useful post for me.
    Devops Training
    Devops Online Training
    DevOps Training institute in Ameerpet

  11. Nice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updating Devops online training

  12. Hi Thanks for sharing the devOps hand book summary post
    DevOps Training in Hyderabad