DevOps Handbook Summary 4 of 4 - Security

Book summary of The DevOps Handbook by Gene Kim et. al. Excerpted content is formatted in italics.

Part VI: The Technical Practices of Integrating Information Security, Change Management, and Compliance

The DevOps goal is to make security a part of everyone's job. We'll look for opportunities to augment our controls with audit-able automation. This automation will minimize the need for separation of duties and change approvals that unnecessarily impede the value chain.  Once automated and baked into everyone's daily work the controls are less variable, more audit-able, and significantly stronger than the manual controls they replace. Some critical controls will remain manual.

We do this by:
  • Making security a part of everyone’s job 
  • Integrating preventative controls into our shared source code repository 
  • Integrating security with our deployment pipeline 
  • Integrating security with our telemetry to better enable detection and recovery 
  • Protecting our deployment pipeline 
  • Integrating our deployment activities with our change approval processes 
  • Reducing reliance on separation of duty

Information security as everyone's job every day

Integrate security:
  • Into development iteration demos - this means they are their own stories or are acceptance criteria of all relevant stories
  • Into defect tracking and post mortem's  - security defects should be tracked with all other defects and security incidents and security implications of any incident are subject to post mortem reviews
We must concern ourselves not only with application and data center security but with end to end value chain security.

Like all of our code be it application, infrastructure, operations etc.  security capabilities reside in our shared code repository standards making them easy to find, understand and use.  We may include items such as:
  • Code libraries and their recommended configurations (e.g., 2FA [two-factor authentication library], bcrypt password hashing, logging) 
  • Secret management (e.g., connection settings, encryption keys) using tools such as Vault, sneaker, Keywhiz, credstash, Trousseau, Red October, etc. 
  • OS packages and builds (e.g., NTP for time syncing, secure versions of OpenSSL with correct configurations, OSSEC or Tripwire for file integrity monitoring, syslog configuration to ensure logging of critical security into our centralized ELK stack)

Next we'll integrate security into our development pipelines by including as many automated security tests as we can to run with all our other automated tests. 

Tools such as Gauntlt have been designed to integrate into the deployment pipelines, which run automated security tests on our applications, our application dependencies, our environment, etc. Remarkably, Gauntlt even puts all its security tests in Gherkin syntax test scripts, which is widely used by developers for unit and functional testing. Doing this puts security testing in a framework they are likely already familiar with. This also allows security tests to easily run in a deployment pipeline on every committed change, such as static code analysis, checking for vulnerable dependencies, or dynamic testing.

Ensure security of the application

Developers, often focussed on happy path tests of correctness, will need security training to ensure use of sad or bad path automated tests and tools such as:
  1. Static analysis tools - Brakeman and Code Climate
  2. Dynamic analysis - Focusses on run time behavior. Tools include: ArachniOWASP ZapNmap and metasploit
  3. Dependency scanning - for malicious or vulnerable binaries
  4. Source code integrity and signing  - all developers are identified and use a security key e.g. PGP, all packages generated by continuous integration should be signed and inventoried for audit-ability.

Ensure the security of our environments

In this step, we should do whatever is required to help ensure that the environments are in a hardened, risk-reduced state. Although we may have created known, good configurations already, we must put in monitoring controls to ensure that all production instances match these known good states. 

We do this by generating automated tests to ensure that all appropriate settings have been correctly applied for configuration hardening, database security settings, key lengths, and so forth. Furthermore, we will use tests to scan our environments for known vulnerabilities.

Another category of security verification is understanding actual environments (i.e., “as they actually are”). Examples of tools for this include Nmap to ensure that only expected ports are open and Metasploit to ensure that we’ve adequately hardened our environments against known vulnerabilities, such as scanning with SQL injection attacks. The output of these tools should be put into our artifact repository and compared with the previous version as part of our functional testing process. Doing this will help us detect any undesirable changes as soon as they occur.

Incorporate security into telemetry

Environmental examples:
  • OS changes (e.g., in production, in our build infrastructure) 
  • Security group changes 
  • Changes to configurations (e.g., OSSEC, Puppet, Chef, Tripwire) 
  • Cloud infrastructure changes (e.g., VPC, security groups, users and privileges) 
  • XSS attempts (i.e., “cross-site scripting attacks”) 
  • SQLi attempts (i.e., “SQL injection attacks”) 
  • Web server errors (e.g., 4XX and 5XX errors)
Application examples:
  • Successful and unsuccessful user logins 
  • User password resets 
  • User email address resets
  • User credit card changes

Protect our development pipeline

Example countermeasures:
  • Hardening continuous build and integration servers and ensuring we can reproduce them in an automated manner
  • Reviewing all changes introduced into version control
  • Instrumenting our repository to detect when test code contains suspicious API calls 
  • Ensuring every CI process runs on its own isolated container or VM 
  • Ensuring the version control credentials used by the CI system are read-only

Protect our deployment pipeline

Integrate security and compliance into the change approval processes

Change management processes typically these address three types of changes:
  • Standard - low risk, maybe pre-approved
  • Normal - higher risk, typically requiring multiple party review
  • Urgent - high risk, often requiring executive approvals
Our goal is demonstrate that as a result of all of the automated and manual controls we have in place that a large majority of changes are standard changes and similarly that many urgent changes may be treated as normal changes.

Reduce reliance on separation of duty controls

When we did production deployments less frequently (e.g., annually) and when our work was less complex, compartmentalizing our work and doing hand-offs were tenable ways of conducting business. However, as complexity and deployment frequency increase, performing production deployments successfully increasingly requires everyone in the value stream to quickly see the outcomes of their actions. 

Separation of duty often can impede this by slowing down and reducing the feedback engineers receive on their work. This prevents engineers from taking full responsibility for the quality of their work and reduces a firm’s ability to create organizational learning. 

Consequently, wherever possible, we should avoid using separation of duties as a control. Instead, we should choose controls such as pair programming, continuous inspection of code check-ins, and code review. These controls can give us the necessary reassurance about the quality of our work. Furthermore, by putting these controls in place, if separation of duties is required, we can show that we achieve equivalent outcomes with the controls we have created.

To accomplish this we need to ensure we have documentation and proof for auditors and compliance officers.

As technology organizations increasingly adopt DevOps patterns, there is more tension than ever between IT and audit. These new DevOps patterns challenge traditional thinking about auditing, controls, and risk mitigation. 

As Bill Shinn, a principal security solutions architect at Amazon Web Services, observes, “DevOps is all about bridging the gap between Dev and Ops. In some ways, the challenge of bridging the gap between DevOps and auditors and compliance officers is even larger. For instance, how many auditors can read code and how many developers have read NIST 800-37 or the Gramm-Leach-Bliley Act? That creates a gap of knowledge, and the DevOps community needs to help bridge that gap.”


  1. Everyone wants to get unique place in the IT industry’s for that you need to upgrade your skills, your blog helps me improvise my skill set to get good career, keep sharing your thoughts with us.

    Devops Online Training

  2. That was a nice to read, looking forward to see the next post..Thanks for the information
    More on Devops training

  3. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site"Devops Training in Chennai".

  4. Greeting! Just leaving a note to let you know how much I appreciate this post, I can tell a lot of effort had been put in! Keep it up! If you ever want to register a business, I know the best business incorporation provider! accounting company now!

  5. Thanks for providing such a great information about devops, i need exact details about who providing devops online training.

  6. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information...Devops Training in Chennai
    Devops Training Institute in Chennai

  7. Awesome tips, thank you very much! I will be sharing and recommending this post to my blogging friends. DevOps Course | AWS/Python Training in Bangalore

  8. really cool post, highly informative and professionally written and I am glad to be a visitor of this perfect blog, thank you for this rare info!
    devops training in hyderabad

  9. Excellent Article ...thank u for sharing, such a valuable content Learners to get good knowledge after read this article.. Oracle Training in Chennai | Oracle Training Institute in Chennai

    Oracle Training in Chennai | Oracle Training Institute in Chennai

  10. I feel really happy to have seen your webpage and look forward to so many more entertaining times reading here. Thanks once more for all the details.
    Devops Training in Chennai
    Devops Training Institute in Chennai

  11. Nice information you have shared, It's useful beginners. Please keep updates on Devops Online Training Bangalore

  12. Thanks a lot very much for the high your blog post quality and results-oriented help. I won’t think twice to endorse to anybody who wants and needs support about this area.
    java training in bangalore

  13. Those guidelines additionally worked to become a good way to recognize that other people online have the identical fervor like mine to grasp great deal more around this condition.
    Devops Training in Bangalore

  14. Ciitnoida provides Core and java training institute in noida. We have a team of experienced Java professionals who help our students learn Java with the help of Live Base Projects. The object-oriented, java training in noida , class-based build of Java has made it one of most popular programming languages and the demand of professionals with certification in Advance Java training is at an all-time high not just in India but foreign countries too.

    By helping our students understand the fundamentals and Advance concepts of Java, we prepare them for a successful programming career. With over 13 years of sound experience, we have successfully trained hundreds of students in Noida and have been able to turn ourselves into an institute for best Java training in Noida.

    java training institute in noida
    java training in noida
    best java training institute in noida
    java coaching in noida
    java institute in noida

  15. ERP-SAP-SD Training Centre in Noida

    CIIT Noida provides Best SAP Training in Noida based on current industry standards that helps attendees to secure placements in their dream jobs at MNCs. CIIT Provides Best ERP SAP Training in Noida. CIIT is one of the most credible ERP SAP training institutes in Noida offering hands on practical knowledge and full job assistance with basic as well as advanced level ERP SAP training courses. At CIIT ERP SAP training in noida is conducted by subject specialist corporate professionals with 7+ years of experience in managing real-time ERP SAP projects. CIIT implements a blend of aERPemic learning and practical sessions to give the student optimum exposure that aids in the transformation of naïve students into thorough professionals that are easily recruited within the industry.

    At CIIT’s well-equipped ERP SAP training center in Noida aspirants learn the skills for ERP SAP Basis, ERP SAP ABAP, ERP SAP APO, ERP SAP Business Intelligence (BI), ERP SAP FICO, ERP SAP HANA, ERP SAP Production Planning, ERP SAP Supply Chain Management, ERP SAP Supplier Relationship Management, ERP SAP Training on real time projects along with ERP SAP placement training. ERP SAP Training in Noida has been designed as per latest industry trends and keeping in mind the advanced ERP SAP course content and syllabus based on the professional requirement of the student; helping them to get placement in Multinational companies and achieve their career goals.

  16. BCA Colleges in Noida

    CIIT Noida provides Sofracle Specialized B Tech colleges in Noida based on current industry standards that helps students to secure placements in their dream jobs at MNCs. CIIT provides Best B.Tech Training in Noida. It is one of the most trusted B.Tech course training institutes in Noida offering hands on practical knowledge and complete job assistance with basic as well as advanced B.Tech classes. CIITN is the best B.Tech college in Noida, greater noida, ghaziabad, delhi, gurgaon regoin .

    At CIIT’s well-equipped Sofracle Specialized M Tech colleges in Noida aspirants learn the skills for designing, analysis, manufacturing, research, sales, management, consulting and many more. At CIIT B.Tech student will do practical on real time projects along with the job placement and training. CIIT Sofracle Specialized M.Tech Classes in Noida has been designed as per latest IT industry trends and keeping in mind the advanced B.Tech course content and syllabus based on the professional requirement of the student; helping them to get placement in Multinational companies (MNCs) and achieve their career goals.

    MCA colleges in Noida we have high tech infrastructure and lab facilities and the options of choosing multiple job oriented courses after 12th at Noida Location. CIIT in Noida prepares thousands of engineers at reasonable B.Tech course fees keeping in mind training and B.Tech course duration and subjects requirement of each attendee.

    Engineering College in Noida"

  17. Devops is the process of development and process, Good blog for devops queries thanks for sharing check at Devops Online Training Hyderabad