The one-sentence version
Security incidents will happen sooner or later. When they do, the biggest fear is not the incident itself, but panic: not knowing who is in charge, what to do first, or who to notify, resulting in expanding losses and destroyed evidence. A mature approach is to prepare a workflow in advance: Preparation → Detection & Analysis → Containment → Eradication → Recovery → Postmortem. A postmortem must be blameless (focusing on the event, not the person), aiming to find out "why the system allowed this to happen" rather than "who made the mistake."
This article has three layers:
- Layer 1: The Intuition: Division of labor during a fire
- Layer 2: Real-World Use: NIST framework, Google SRE's blameless postmortem, and good examples of public postmortems
- Layer 3: Principles and engineering key points: What to do in each stage, the specificities of cryptocurrency scenarios, and postmortem templates
Layer 1: The Intuition
1.1 Division of Labor During a Fire
When a fire breaks out, the most effective approach is not for everyone to rush in together, but rather:
- One person commands, deciding what to do first;
- Someone is responsible for extinguishing the fire (containment);
- Someone is responsible for evacuation and notification (communication);
- Someone is responsible for recording what happened (investigation).
Afterward, sit down and calmly analyze: How did the fire start? Why didn't the alarm go off? Why was the fire extinguisher expired?
1.2 Two Common Mistakes
| Mistake | Consequence |
|---|---|
| Rushing to "clean up" as soon as an incident is detected | Deleted logs or reinstalled servers mean evidence is lost, and you won't know how the attacker entered or if they are still present |
| Postmortem turns into a blame game | People start hiding mistakes, meaning future issues will be discovered even later |
Layer 2: Real-World Use
- NIST SP 800-61: The incident handling guide published by the National Institute of Standards and Technology. It is the most frequently cited framework in the industry, dividing incident response into several stages: Preparation, Detection & Analysis, Containment, Eradication & Recovery, and Post-Incident Activity (the latest version further aligns it with the NIST Cybersecurity Framework).
- Blameless postmortem: A practice promoted by Google in its book Site Reliability Engineering. A postmortem assumes that everyone made reasonable decisions given the information available to them at the time, focusing on analyzing flaws in processes, tools, and system design.
- Good examples of public postmortems: After accidentally deleting its production database in 2017, GitLab live-streamed the recovery process and published a detailed postmortem listing all failed backup mechanisms and improvement measures. Companies like Cloudflare also regularly publish detailed incident postmortems. These have become industry learning materials.
- Cryptocurrency field: After a theft occurs, project teams usually release a preliminary statement within hours and a detailed technical postmortem within days or weeks; on-chain security firms also conduct independent analyses. The investigation report for the Bybit incident (see "Bybit Incident and Frontend Integrity") is an example.
Layer 3: Principles and Engineering
3.1 What to do in each stage
| Stage | Key Actions |
|---|---|
| Preparation | Write incident response manuals; define who is the Incident Commander; prepare contact lists (cloud providers, exchanges, stablecoin issuers, lawyers, law enforcement); conduct drills (see "Chaos Engineering and Fault Injection") |
| Detection & Analysis | Confirm if it is truly an incident; determine severity and impact scope; start recording the timeline |
| Containment | Stop losses from expanding: revoke credentials, suspend services, isolate compromised machines, pause contracts. Short-term containment can be aggressive to stop the bleeding first |
| Eradication | Find and remove the attacker's foothold: backdoors, implanted code, stolen credentials; fix vulnerabilities |
| Recovery | Restore services from a clean state; strengthen monitoring to observe if the attacker returns |
| Postmortem | Timeline, root cause, what went well, what went poorly, improvement items (each with an owner and deadline) |
3.2 Preserving Evidence
- Isolate rather than destroy during containment: Disconnect the machine's network instead of directly shutting it down and reinstalling the OS;
- Take snapshots and export logs before attempting repairs;
- Audit logs should be stored where attackers cannot modify them (see "Hash-chain Audit Logs");
- Record every operation: who, what time, and what was done.
3.3 Communication
| Target | Key Points |
|---|---|
| Internal | Designate a communication lead to provide regular updates and prevent rumors |
| Users | Explain as early as possible "what happened, who is affected, and what users need to do"; do not guess if uncertain |
| Regulators | Many jurisdictions have statutory reporting timelines for data breaches |
| Partners | Cloud providers, exchanges, stablecoin issuers (in crypto incidents, the earlier you contact them, the higher the chance of freezing funds) |
3.4 Specificities of Cryptocurrency Incidents
- Time is extremely critical: Funds can be transferred, swapped for other coins, or sent to mixers within minutes. Containment must be measured in minutes;
- Immediate actions: Pause contracts (if a pause function exists), revoke authorizations, transfer remaining funds to a secure address, contact stablecoin issuers and major exchanges with the attacker's address (see "On-chain Tracking and Fund Freezing");
- Security emergency organizations: Groups like SEAL 911, composed of white-hat hackers, can provide rapid assistance during an incident;
- Communicating with the attacker: Many projects leave messages on-chain, offering terms such as "return most of the funds and keep a portion as a bounty" (see "Responsible Disclosure and Bug Bounties").
3.5 Postmortem Template
## Summary
One or two sentences: what happened, the impact, and the current status.
## Timeline (use a single time zone)
- Time — Event (discovery, decision, action)
## Root Cause
It's not "someone made a mistake," but "why the system allowed this mistake to cause such consequences."
You can ask "why" five times in a row.
## What went well
## What went poorly
## Luck factor
Which parts were just lucky? Next time might not be as lucky.
## Action Items
| Improvement | Owner | Deadline | Status |
The "Luck Factor" section is often overlooked but is very important: it reveals the weak links that did not cause an incident this time, but might next time.
Summary
| Layer | Key points |
|---|---|
| Process | Preparation → Detection & Analysis → Containment → Eradication → Recovery → Postmortem |
| Key | Stop the bleeding first; isolate rather than destroy to protect evidence; one person commands, one person communicates |
| Cryptocurrency | Measured in minutes; contact issuers and exchanges as soon as possible; use security emergency organizations |
| Postmortem | Focus on the event, not the person; list the luck factor; improvement items must have owners and deadlines |