Fail-closed: Better to Stop Than to Let Through When Errors Occur

This article has three layers:

1. Layer 1: The Intuition: Should electronic locks open or stay locked when the power goes out?
2. Layer 2: Real-World Use: Firewalls, antivirus software, circuit breakers, and blacklist updates.
3. Layer 3: Principles and engineering key points: Identifying "errors," protecting data sources, automatic loss prevention, and the cost of availability.

Layer 1: The Intuition

1.1 Power outage in an electronic lock

If the electronic access control system in a building loses power, the doors should:

• Open automatically (fail-open / fail-safe): So people can escape during a fire. Fire safety regulations usually require escape routes to be designed this way.
• Stay locked (fail-closed / fail-secure): For vaults, server rooms, or prisons, people should not be able to enter or exit freely just because the power is out.

There is no absolute "correct" answer; it depends on what you fear most when an error occurs: fear of people being unable to get out, or fear of bad actors getting in.

1.2 The same problem in software

def is_allowed(address):
    try:
        return address not in fetch_blocklist()
    except NetworkError:
        return True   # ← fail-open: if the list cannot be fetched, assume it is fine

This code looks "friendlier": the user can continue operating even during network jitter. However, if an attacker finds a way to make the blacklist service unreachable (e.g., by attacking it or interfering with the network), all checks become ineffective.

Changing to fail-closed:

    except NetworkError:
        return False  # If it can't be checked, assume it failed

Layer 2: Real-World Use

The OCSP example illustrates: fail-closed is not free; it shifts the availability pressure onto the dependent services.

Layer 3: Principles and Engineering

3.1 First, identify "errors"

The most insidious form of fail-open is not except: return True, but treating an error as a normal empty result:

Therefore: You must explicitly distinguish between "successfully obtaining an empty result" and "failing to obtain a result"; the latter must trigger an explicit error.

3.2 Protection for data source updates

External lists (fraudulent addresses, sanction lists, malicious domains) need regular updates. The update process itself can be attacked: an upstream repository could be poisoned to push an almost empty list; if the downstream automatically accepts it, all checks become useless.

Protective measures:

• Refuse to replace the old list if the new list is significantly smaller; keep the old copy and trigger an alert;
• If a fetch fails, continue using the previous local copy and display a message like "List has not been updated for [X] days";
• Treat an empty list as an error, not as "there are no bad actors."

3.3 Stop when inconsistent

In multi-source cross-verification, when two independent sources provide different answers, the correct reaction is to stop, rather than picking the one that looks more convenient. This is the application of fail-closed in redundant systems.

3.4 Automatic loss prevention

A more aggressive form of fail-closed is: when the system detects it might have been compromised, it proactively shuts down services. For example, if the frontend page does not match the expected state, automatically take the page offline—it is better for everyone to be temporarily unable to use it than to allow anyone to sign something on a potentially tampered page (see The Bybit Incident and Frontend Integrity).

Design key points:

• The credential to execute a shutdown should only have the "shutdown" permission (see Least Privilege and Key Management); the worst-case scenario of a stolen credential is service interruption, not data tampering;
• After a shutdown, human confirmation must be required to restore service;
• Prevent frequent shutdowns caused by false positives; for example, network jitter should not trigger a shutdown, only content mismatches should.

3.5 Cost and balance

Fail-closed makes a system "brittle": if any single dependency jitters, the user cannot use the system. Ways to balance this:

• Distinguish risk levels of decisions: Read-only displays can use degraded modes, but operations involving money must be fail-closed;
• Provide redundancy for dependencies: The probability of two independent data sources failing simultaneously is much lower than one;
• Be clear about failures: Tell the user "This operation has been paused for security reasons because the [X] check is temporarily unavailable," rather than providing a vague error;
• Monitor failure rates: A sudden spike in fail-closed rejections is itself a signal worth investigating.

Summary