Doing SecOps Right - Runbooks

The Case for Runbooks

Security operations is the fire fighting work of blue team security. Ops is the grunt work on the front line, in the trenches, where you ingest and triage security alerts from automated systems, users, and security appliances. The operations team is often the first to spot the smoke that could be indicative of the fire that forms the next big organizational breach and as such it is critical that the team is equipped with the tools and resources to work efficiently, effectively, and consistently.

Unfortunately in most organizations the operations components of security, from monitoring and detection all the way through incident response, is often present only as tribal knowledge or individual expertise. Practitioners often follow wildly different processes depending on their skill level, knowledge, and professional experience. This can lead to wide variety in quality when not only detecting, but also responding to a compromise.

Rather than rely on an ad hoc process or concede that all security events are different and have to be addressed individually, mature organizations develop written procedures for how to handle security event in the same way that they have procedures for IT operational procedures. By having a written procedure it becomes possible to leverage the power of a checklist to ensure that process is consistent. Even poorly devised procedures provide, at the very least, consistency in execution. Ideally procedure documents are evolutionary and can be improved over time. By writing process down you also develop the opportunity to cement practices and retain information about what works when following a process, and the ability to winnow out approaches that prove ineffective over time.

Having security runbooks (also called playbooks) keeps quality consistent as well, since each member of the security team approaches process in the same manner. This allows knowledge transfer too, as the more experienced practitioners update the runbooks with their insights allowing the more junior members of the team to leverage the experience of their peers. Runbooks also allow you to easily onboard new members of the team since they can follow the runbook right away and hopefully reproduce a minimum quality in performing a task.

Where to Keep Runbooks

Ideally your security runbooks are living documents. As such a format like a file stored on a shared drive or on a document repository is a poor choice. It can work, but becomes burdensome when revisions are being made, copies are modified, or multiple people need to access and edit the runbook.

The best way I have found to store runbooks is on a wiki. The wiki system was designed specifically for revision tracking, authorship attestation, concurrency, and easy access. Wikis are like digital whiteboards and theyre perfect for evolutionary documentation like runbooks.

Wiki also allows you to cross reference and link from one runbook to another. This makes for easy navigation and allows you to subdivide runbooks into smaller, self contained process documents. This allows you to make modular runbooks that can be reused (in much the same way that programmers develop functions for duplicative processes).

Format

Your runbooks should have a consistent format. They should have a section up front defining scope, definitions, roles and responsibilities, and authority. The runbook should have a main section that covers the process to follow. This section could include process flow charts as well for a quick visual reference. The runbook should include all the details that a practitioner needs and might include links to systems, screenshots, and references.

Your runbooks might also contain additional sections for things like communications plans. This could outline how, and when, to send messages to other teams, customers, or stakeholders. You can also include sample reports, communications, or emails. This should also cover the type of system used for communications, whether its a ticketing system, email, or a phone call. The communication plan helps to ensure that hand offs between the security operations team and other parties are consistent and reliable.

Runbooks should also outline any records, reports, or metrics that should be captured during the course of the security event. Metrics are vital to continuous improvement and measurement. Defining metrics as part of the runbooks ensures that theyre collected accurately and consistently.

Adoption and Revision

Whenever changes are made to the runbooks the entire security operations team will need to review the changes and be educated about them. This applies to totally new runbooks as well. Whomever writes the runbook should gather the team and explain the runbook to them, take questions, and address any concerns.

At the conclusion of any security event, it is vital to perform an after action review. Note and strengths or deficiencies in the defined process and update runbooks accordingly. As soon as new tactics, approaches, or strategies are developed add them to the runbook. Similarly if you find that parts of the runbook are stale, never get used, or arent helpful they should be stricken from the runbook.

Types of Runbooks

One area of concern with organizations new to developing runbooks is what sorts of runbooks to write. There are numerous online resources to help you get started, but the best predictor of future security events are the past. Look back historically and try to discern what types of security events are most likely to occur next. Another approach is to talk to peers or examine open source outlets to determine what types of incidents are occurring in the wild. A third approach could be to use threat models to determine what type of malicious actors and actions could be likely to take place in your organization. When all else fails its always a good idea to look at the VERIS framework (http://veriscommunity.net/) for guidance.