Building a Purple Team Program

Purple Teaming, also known as adversary emulation, is an increasingly popular way to proactively test security defenses. Organizations looking to justify their investment in security tools seek ways to validate that tools work as expected and that settings are sufficient to provide protection against attacks. A purple team program can accomplish these goals by simulating attack activity in an open forum during which blue team security defenders can monitor tools like Endpoint Detection and Response (EDR) and Security Information and Event Management (SIEM) tools.

Starting a purple team program can seem like a daunting task, and many programs get hung up on tools and technology without realizing the importance of process and governance. In order to support a successful purple team program, organizations must first establish success criteria and develop methods of measuring and reporting on the program and its effectiveness. A purple team program without strong governance and reporting structure, as well as formal process, is bound to struggle to be effective.

Sponsorship

The first thing a purple team program needs is sponsorship and governance. Someone, or some role, must assume responsibility for the program. Without accountability it is far too easy for a program to wither and die after just one or two iterations of testing. By formalizing ownership of a program, and tying program outcomes to personal goals and objectives, and performance evaluation, organizations can ensure commitment to the program and its goals. It is important to tie program execution and improvement to the specific performance objectives of individuals so that they are rewarded for a well executed program and incentivized to prioritize the program in the face of competing demands.

Governance

Once a responsible party is identified to own the purple team program it becomes necessary to outline the goals and objectives of the program itself. Developing a policy on purple teaming can serve as an important guidepost to articulate what the program is, and importantly is not, designed to accomplish. The policy should state the scope and cadence of purple team exercises, the roles and responsibilities of various parties involved in the program and the intent of the program itself.

Process

Closely following a stated policy around purple teaming, organizations should develop a formal process for purple teaming. Process documentation should cover a high level detail of how the actual testing should be carried out. This process should cover the testing lifecycle from planning to reporting.

Purple team exercises should include development of a collaborative backlog of planned testing scenarios that can draw on emerging threats, vulnerabilities, and inputs from other teams such as incident response, threat intelligence, and security operations. By drawing on observations of various teams, including concerns from executive leadership, the purple team program can propose timely tests that are relevant to the business needs.

As the backlog is developed, the process documentation should include a mechanism for evaluation and selection of testing scenarios. Typically this takes place at a scheduled meeting that includes relevant stakeholders. By extending an opportunity to provide input to other teams the purple team democratizes the process and encourages ownership of outcomes across the organization. Of particular interest will be the security operations team as well as adjacent security and leadership teams. These representatives should discuss options for scenarios and agree on which backlog items to prioritize and which to defer. In general the backlog should be ranked on impact and prevalence. Tests that evaluate high impact risks that could affect a broad, or important, swath of the organization should be prioritized over more niche tests that affect small or less important segments of the organization.

Once a testing scenario is decided upon, the purple team should develop a formal proposal for testing. This proposal should outline the testing scenarios, say resilience to malicious email attachments, and describe the benefits and intent of testing. The proposal should describe systems that will be tested or impacted, mitigation measures, and also include any necessary sign off on risk. The proposal should include a testing schedule, communications plan to inform relevant stakeholders, as well as contingencies for adverse findings in the case where the test reveals a critical vulnerability. In general, in these circumstances, the proposal should outline how the test will be paused and the critical vulnerability or adverse finding will be addressed. It is also important that the proposal describe any risks to the test, such as competing priorities. For instance, if the purple team includes members of the incident response team (IR), testing might become delayed due to higher priority responsibilities of the IR team to respond to events.

Team Composition

In order to maximize effectiveness, the purple team should be composed of at least one member with red team, or penetration testing, experience and one member with extensive blue team, or cybersecurity defensive experience such as security operations or incident response. The blended red and blue sides create an effective purple team because it is necessary to understand attacker tools, techniques, and procedures (TTPs) as well as defensive capabilities. A purple team exercise is a collaboration between these two parties, with one helping the other. This stands in contrast to a traditional penetration test or red team exercise, where the parties are adversarial. In an effective purple team each party works with the other to enhance the test and to evaluate security controls.

Testing

Once proposals are developed the purple team should go about scoping the tests, deciding on which MITRE ATT&CK techniques to simulate, tools to be used, and tracking and reporting technologies to be employed. In many ways, this is the simplest part of purple team testing. An effective purple team test can be conducted with just access to a representative workstation for the organization, PowerShell, and a spreadsheet to track the test.

Tracking should include the date and time of the test, the source and destination of the test, description of the tools used and notes for any special arguments or circumstances. It is important to track tests so that if no detection or alert is generated the teams can use the log of activity to trace back sources or evidence of tests. A comprehensive log also allows for tests to be repeated in the future in order to test any improvements or modification to the environment.

As tests are run, blue team defenders should monitor SIEM, EDR, and other security tools for any sign of the test being run. Tests should be evaluated on three axes for success or failure:

1. Whether the test activity was blocked by security tools or configuration
2. Whether the test resulted in any logs being produced
3. Whether an alert was raised to monitoring staff

These axes are important to understand for various reasons. If a test is blocked that is a positive outcome, but if no alert is raised to monitoring then the event is invisible to cybersecurity staff and an opportunity to investigate adversary activity could be missed. If a simulated attack test is successful and a log is created in a system but no alert is raised this is a crucial finding, because it means that a detection could be created in a monitoring system and the test outcome could be improved. Without any logging of testing activity, however, there is a gap in security visibility and no amount of detection engineering can mitigate the simulated attack. Reporting on blind spots in logging and alerting capabilities is a critical outcome of purple team testing. Finally alerting is important because it represents an organization's best chance to respond to attacker activity. Even if an attack is blocked it is important that an alert be raised and an investigation conducted. Alerts should also be evaluated for appropriateness. For instance, if an attack activity represents a critical risk, such as an attempt to dump LSASS, a correspondingly critical alert should be produced. A mismatch of priority between an attack activity and an alert could mean that an investigation is mis-categorized or an investigative opportunity overlooked.

In general, it is important to develop tests that can be repeated in an automated fashion. For this reason using testing frameworks such as Atomic Red or MITRE Caldera can be very useful. Once an initial test is run, automation can allow a test to be reproduced after mitigation or configuration changes to measure the impact of these adjustments on the overall risk exposure of an organization. While automation and repetition are hallmarks of a mature program, they are not necessary for a successful initial purple team program.

Reporting

The reporting phase of a purple team is where the real value of the program shines. Tests should be described in detail in the report and suggestions for improvement and mitigation should be made. Ideally these recommendations can be incorporated into formal project plans that can be tracked and evaluated over time. Purple team testing is designed to help organizations evaluate their risk, but more importantly to improve their posture. Formal reports should be written, delivered to key stakeholders and executives, and filed in a document archive for future reference.

Program Maintenance

A purple team should include a leadership component responsible for logistics, resourcing, and reporting. This party should ensure that tests are carried out on schedule and that reports are compiled and delivered to leadership. An ongoing purple team will discover needs for resources and leadership should carry these resource requests to business leadership armed with reports and outcomes to justify value of the purple team. Ideally the purple team becomes an embedded component of the broader cybersecurity program and is aligned to other ongoing initiatives such as detection engineering, security operations, or cyber threat intelligence.

Conclusion

Developing a robust purple team program doesn't necessarily require extensive investment in tools or people. A purple team program can provide an opportunity for existing staff to extend their capabilities, learn new skills and technologies, and contribute in new ways to a program. Some of the most difficult challenges to developing a purple team program are non-technical, such as the development of policy, process, and governance. Once these components are in place the technical tools can begin simply and scale as the program matures.