The one-sentence version
The two most common failures of a monitoring system are: first, too many alerts: hundreds of messages every day, most of which are false positives, causing the truly important one to be buried; and second, the monitoring system itself breaks without anyone knowing: no alerts are sent, everyone assumes everything is normal, but in reality, the monitoring has long since stopped working. A good alert only notifies people of things that "require immediate action" and is categorized by severity; meanwhile, use heartbeats to prove the monitoring itself is still alive: "no news" does not equal "good news."
This article has three layers:
- Layer 1: The Intuition: The Boy Who Cried Wolf and the Sleeping Sentry
- Layer 2: Real-World Use: The Target and Equifax data breaches
- Layer 3: Principles and Engineering Key points: What is worth alerting, grading and routing, heartbeats, and reducing false positives
Layer 1: The Intuition
1.1 Two Fables
| Fable | Monitoring Equivalent |
|---|---|
| The Boy Who Cried Wolf: Cried too many false alarms, so no one listened when the real wolf arrived | Too many alerts, too many false positives; people learn to ignore them |
| The Sleeping Sentry: The sentry fell asleep, and the camp thought they were safe all night | The monitoring program crashed or the alerting channel broke; no alerts are received, which is mistaken for "everything is normal" |
The result of both situations is the same: When a real incident occurs, no one takes action.
1.2 Standards for a Good Alert
A good alert should simultaneously satisfy these three criteria:
- Requires human action (otherwise, it is just a log entry);
- Requires action now (otherwise, it can be put into a daily report);
- The recipient knows what to do (otherwise, it only creates anxiety).
Layer 2: Real-World Use
- Target (2013): Attackers entered Target's network through the credentials of an HVAC vendor, implanted malware in the point-of-sale systems, and ultimately stole information from approximately 40 million payment cards. Post-incident investigations showed that the malware detection system deployed by Target did indeed issue an alert and forwarded it to the security team, but no one took action. This has become the most frequently cited case of "alert fatigue."
- Equifax (2017): Personal information of approximately 147 million Americans was stolen. Investigations revealed that a monitoring device used to inspect encrypted traffic had an SSL certificate that had expired 19 months prior; during this time, it was completely unable to see the content within the encrypted traffic. After the certificate was updated, the team discovered suspicious traffic almost immediately. This is a classic case of "the sentry fell asleep."
- Common Phenomenon: Long-term surveys in the security industry show that security operations teams face a massive volume of alerts every day, a significant proportion of which are never investigated. High false positive rates are one of the primary pain points for teams.
Layer 3: Principles and Engineering
3.1 What is Worth Alerting
| Worth Alerting Immediately | Put in Logs or Daily Reports Only |
|---|---|
| Production page content does not match expectations | A single failed login |
| An unregistered deployment version appears | A single request timeout |
| Hash chain breakage in audit logs (see "Hash Chain Audit Logs") | Disk usage at 70% |
| Administrator account logging in from an unfamiliar location | A normal deployment completion |
| Inconsistent results from multiple data sources causing a transaction rejection | Scanned traffic that was normally intercepted |
| Login failure rate is 10x the normal rate (see "Rate Limiting and Abuse Resistance") |
Principle: Alerts should target symptoms and consequences (users are affected, security invariants are broken), rather than every fluctuation in low-level metrics.
3.2 Grading and Routing
| Level | Meaning | Channel |
|---|---|---|
| Critical | An ongoing security incident or financial risk | Phone / Push notification; must wake people up even at midnight |
| High | Needs to be handled today | Instant messaging, email |
| Low | Needs attention, can be scheduled | Tickets, daily reports |
Other considerations:
- Deduplication and Aggregation: If the same issue triggers 100 times, send only one alert and include the count;
- Notify on State Change: Alert when transitioning from "normal" to "abnormal"; during a continuous abnormal state, remind at fixed intervals; notify once more when it recovers;
- Jitter Protection: Do not alert for a single occasional network failure; only alert after several consecutive failures; however, strong signals like content mismatch should trigger an alert immediately.
3.3 Heartbeat: Proving the Sentry is Awake
| Method | Description |
|---|---|
| Periodic Heartbeat Messages | The monitoring system actively sends an "I am alive" message at regular intervals; if none is received by the deadline, it indicates a problem with the monitor |
| Dead man's switch | An independent external service waits for a heartbeat and alerts if a timeout occurs |
| Canary | Deliberately trigger a known event that should cause an alert (e.g., accessing a honeypot address) to confirm the alert actually arrives |
| Certificate and Credential Expiry Monitoring | The Equifax lesson: If the monitoring tool's own certificates or tokens expire, it will become "blind" without notice |
The key is that the entity checking the heartbeat must be independent of the system being checked; otherwise, both might fail simultaneously.
3.4 Reducing False Positives
- Every alert rule must have an owner and a runbook (what to do as the first step upon receipt);
- Regular reviews: Which alerts from last month were never handled? Either modify the rule or delete it;
- False positives are also data: Every time a false positive is marked, it serves as a basis for adjusting thresholds;
- New rules should run in a "log-only, no-alert" mode for a period to observe the false positive rate.
3.5 After the Alert
An alert is just the beginning. The workflow following a critical alert (who is responsible, how to judge, how to mitigate loss, how to communicate) belongs to incident response; see "Incident Response and Post-Mortem."
Summary
| Layer | Key points |
|---|---|
| Two Failures | The Boy Who Cried Wolf (alert fatigue, Target), The Sleeping Sentry (monitoring blindness, Equifax) |
| Good Alert | Requires action, requires it now, knows what to do |
| Engineering | Grading and routing, deduplication and aggregation, state change notification, jitter protection |
| Heartbeat | Independent dead man's switch; monitor the monitor's own certificates and credentials |