The one-sentence version
Systems will always fail: servers crash, network latency occurs, dependent services return errors, or data sources return fake data. Instead of waiting for it to happen in the middle of the night, it is better to proactively create faults under controlled conditions to see how the system reacts. This is Chaos Engineering. For security, it answers the question: Will my designed fail-closed mechanisms, cross-validation, alerting, and automatic loss prevention actually work when a real failure occurs? Many lines of defense are only proven to be functional after they have been "hit" once.
This article has three layers:
- Layer 1: The Intuition: Fire drills
- Layer 2: Real-World Use: Netflix's Chaos Monkey, Game Days, and applications in the security field
- Layer 3: Principles and engineering key points: Steps of an experiment, what faults to inject, and how to control the blast radius
Layer 1: The Intuition
1.1 Fire Drills
A building is equipped with smoke detectors, sprinkler systems, emergency lights, and escape routes. But:
- Are the detector batteries dead?
- Are the sprinkler pipes blocked by construction workers?
- Does everyone know where the escape routes are?
Without drills, you will never know. If you only discover these problems when a real fire breaks out, it is too late.
1.2 From "Hoping it works" to "Knowing it works"
| Design without testing | Proactive fault injection |
|---|---|
| "The system will reject requests when data sources are inconsistent" | Actually make one data source return fake data to see if the system rejects it |
| "An alert will trigger if the page is tampered with" | Actually modify a page in the test environment to see how long it takes for the alert to arrive and to whom it is sent |
| "The system will automatically failover if the primary database crashes" | Actually shut down the primary database |
| "Backups can restore the system" | Actually perform a restoration from a backup |
Layer 2: Real-World Use
- Chaos Monkey (Netflix, since 2011): After Netflix migrated to the cloud, they wrote a program that randomly shuts down server instances in the production environment during working hours, forcing every team to design services such that "it doesn't matter if any single machine disappears at any time." It later expanded into the "Simian Army," capable of simulating entire Availability Zone or even entire region failures.
- Game Day: Teams schedule a specific time to intentionally create faults, with everyone online to observe and respond, followed by a post-mortem. Many large cloud service providers practice this.
- Cloud Provider Fault Injection Services: Major cloud platforms provide managed fault injection tools that can simulate instance termination, network latency, and API errors within a controlled scope.
- Backup and Recovery Drills: During a 2017 GitLab incident, an engineer accidentally deleted the production database, only to find that various backup mechanisms were not working properly. They could only recover from a single, luckily existing snapshot, resulting in a loss of approximately 6 hours of data. They live-streamed the recovery process, and this incident became a classic case study for "backups must undergo regular recovery drills."
Layer 3: Principles and Engineering
3.1 Steps of an Experiment
- Define Steady State: What are the observable metrics when the system is operating normally? (e.g., request success rate, transaction confirmation time, number of alerts)
- Formulate a Hypothesis: "When X occurs, the steady state remains unchanged" or "When X occurs, the system will enter a safe state and trigger an alert within Y minutes";
- Inject Fault: Create X within a controlled scope;
- Observe: Does the steady state hold? Is the alert triggered? Does the system enter the expected safe state?
- Post-mortem and Fix: Areas where the hypothesis fails are the areas that need fixing.
3.2 Security-Related Fault Injection
| Injected Fault | Defense being verified |
|---|---|
| A data source returns erroneous or forged data | Whether multi-source cross-validation rejects it and triggers an alert |
| External blacklist fetch fails or returns an empty list | Whether fail-closed preserves the old list and triggers an alert |
| Page content is modified | How long it takes for integrity checks to detect it (see "Bybit Incident and Frontend Integrity") |
| Transaction parameters are tampered with in the signature chain | Whether the signature verification catches the discrepancy |
| The alerting channel itself fails (e.g., email service is down) | Whether a heartbeat mechanism detects that "the alerting system has gone silent" (see "Security Monitoring and Alert Fatigue") |
| Certificate expiration, clock skew | Whether the system provides a clear error instead of silently degrading |
| Credentials are revoked | Whether the service fails safely instead of falling back to a weaker authentication method |
3.3 Controlling the Blast Radius
| Measure | Description |
|---|---|
| Do it in the test environment first | Consider the production environment only after becoming proficient |
| Start small | Affect only one machine or a small portion of traffic first |
| Have an "Emergency Stop" button | Be able to abort the experiment at any time to restore normalcy |
| Avoid high-risk periods | Do not perform experiments during major promotions or on release days |
| Notify relevant personnel | At least let the on-call staff know to avoid treating it as a real incident |
3.4 Automation
Once mature, fault injection can be integrated into continuous processes:
- Automatically run a set of "fault scenarios" after every deployment to verify that defenses remain effective;
- Perform regular (e.g., monthly) recovery drills: restore from backup, rotate a key, or perform a primary-secondary switchover.
3.5 Mindset
The goal of Chaos Engineering is not "to cause destruction," but to use controlled small faults to avoid uncontrollable large-scale disasters. Every problem discovered during an experiment is a free lesson learned before a real incident occurs.
Summary
| Layer | Key points |
|---|---|
| Intuition | Fire drills: Without drills, you don't know if your defenses are just for show |
| Method | Define steady state → Hypothesis → Inject → Observe → Fix |
| Security Use | Verify if fail-closed, cross-validation, alerting, and backup recovery actually work |
| Control | Test environment first, start small, emergency stop, notify relevant personnel |