Learning From Incidents and Disasters

When something goes wrong, the best course of action (once you have recovered) is to do some reflection and try to identify changes that can be made to systems, processes, or situations to avoid the same thing happening again.

A post-incident review is a structured exercise designed to review the chain of events surrounding an incident or event. By evaluating the activities that led to and resulted from an incident, the post-incident review is able to establish a timeline of events and identify any areas for improvement.

When structured well, a post-incident review is a blameless tool for evaluation, feedback, and process improvement. You can learn more about blameless approaches to post-incident reviews by checking out Etsy’s work in this space.

Holding a Post-Incident Review

A post-incident review should be held after every incident, preferably within two weeks of the main event. This ensures that things are still fresh in people’s minds and that you don’t end up reviewing one incident while handling another.

Everyone involved with an incident should be included in the post-incident review. This may include representatives from external stakeholders and customers where appropriate. A high-level summary of lessons learned and changes made should be added to the customer view of the incident documentation.

Documenting Incidents and Disasters

All incidents should be documented. This documentation serves as a historical record of the incident and the activities resulting from it.

Documentation should contain at a minimum:

• a timeline of events

• example notifications and alerts that triggered the event

• communications sent from and received by the incident response team.

This documentation is useful for audits, and when faced with similar incidents or disasters in the future. It’s always easier to handle a situation if you have the notes of how it was handled last time.

​important​ Be prepared, you may be required to provide a summary of this documentation for distribution to customers, with sensitive details redacted.

While this is rare, remember that your customers are conscious of the risks when using your products and services, and they may choose to request further information if they think the risk has changed.

Common Incident and Disaster Response Pitfalls and How to Avoid Them

Whether you are planning to respond to incidents or disasters, there are a few common challenges and mistakes that companies make. Check out this list and make sure you and your team don’t fall into the same traps.

• Downloading a template and not customizing it to your environment. An auditor comes by one day and does some snooping around. They ask where your incident response plan is and you look sheepishly for an exit, quickly downloading a template from the internet, and passing it over for review.

  We’ve all done it. I don’t judge, but using a template that wasn’t built for your team can be more distracting and dangerous than helpful when faced with a real event.

  Your plan doesn’t need to be fancy. There is no prize for design or how many syllables you use per word. An ugly, misspelled plan that is built for your team, systems, and environment with realistic scenarios is perfect.

• Not testing your plan in a realistic range of scenarios. No matter how young or old your company is, there are many, many ways that an incident or disaster can unfold. Some of them happen to all companies at some point, whereas some are very specific to what your company does.

  For example, a fire is a normal disaster scenario in office buildings, but a chemical spill would be a disaster scenario only found in companies handling hazardous chemicals.

  No matter what your business is, it’s crucial that you list all the possible incident and disaster scenarios you could face and test your plan and playbooks for each of them. While it’s unlikely you will do this all at once, having a test every couple of months, each covering a new scenario, can get you a very long way to being prepared for anything.

• Not including important stakeholders in your tests. Incident response and disaster recovery are definitely team sports. If you find yourself testing a plan on your own in an empty conference room, it’s likely that when the time comes to actually respond to an incident or disaster, the people you need at your side won’t have a clue what to do.

  Testing a plan isn’t just about checking the plan is accurate and works, it’s also a form of collaboration and knowledge sharing. It teaches the team how to work together in the event of something bad happening and what each person needs to do.

  So before you schedule a test all on your own, make sure you list and invite everyone who would have a part to play in the scenario you have chosen to test.

• Failing to document your tests and feed lessons learned back into your processes. Firstly, let’s accept a universal truth. The first plans you write will be wrong. They won’t work or you will find that you made assumptions about the people, systems, and processes that they relate to. Testing our plans is more about finding those flaws and assumptions than it is about proving how good at writing plans you are.

  With this in mind, remember that every issue you identify or assumption you challenge needs to be recorded and fed into your systems and processes. They need to be addressed so that the next test (or real event) doesn’t suffer the same challenges.

  Take notes throughout your testing session, either by having a dedicated scribe or by recording the session for transcription later. Raise any questions or issues into your ticketing system afterwards and ensure they are actioned within 30 days. Make sure the team understands why this is important and are held accountable for this process completing.

Whether your company experiences a small, contained incident or a full-blown disaster, having a well-rehearsed and documented plan makes a huge difference. Make sure your team has both an incident response and disaster recovery plan in place, that they understand how to follow them, and that they challenge any assumptions they are based on.