Race Conditions

Race Conditions

When security depends on the order of operations but the system does not enforce that order, attackers exploit the gap between check and use -- turning microsecond timing windows into privilege escalation, double-spend, and data corruption

When to Use

• Designing financial transactions, balance checks, or any operation where concurrent requests could produce incorrect results
• Reviewing code that checks a condition and then acts on it without holding a lock
• Building file operations that create, read, or write files based on prior checks
• Implementing coupon redemption, vote counting, inventory management, or any system where duplicate processing has business impact
• Choosing database isolation levels and locking strategies for concurrent access
• Reviewing authentication or authorization flows for time-of-check-to-time-of-use gaps

Threat Context

Race conditions are among the most underestimated vulnerability classes because they are non-deterministic -- they may not appear in testing but are reliably exploitable by an attacker who controls timing:

• Ethereum DAO hack (2016): A reentrancy race condition in the DAO smart contract allowed the attacker to recursively withdraw funds before the balance was updated. The attacker drained 3.6 million ETH (approximately $50 million at the time). The vulnerability existed because the contract sent ETH to the caller before updating the caller's balance -- the caller's receive function re-entered the withdraw function and repeated the withdrawal against the stale balance.
• Dirty COW (CVE-2016-5195): A race condition in the Linux kernel's memory management subsystem allowed an unprivileged local user to gain write access to read-only memory mappings. The race existed in the copy-on-write (COW) mechanism -- by racing two threads (one faulting on the mapping, one calling madvise(MADV_DONTNEED)), the attacker could write to files they had only read permission for, including /etc/passwd. This privilege escalation vulnerability existed in the Linux kernel for 9 years before discovery.
• Web application double-spend: An e-commerce platform allowed customers to apply a discount code. Two concurrent requests, each checking that the code had not been used and then marking it as used, both read the code as unused and both applied the discount. At scale, automated tools send hundreds of concurrent requests to exploit such windows.
• Symlink race in temporary files: A privileged program checks that a temporary file does not exist, then creates it. Between the check and the creation, the attacker creates a symlink at that path pointing to /etc/shadow. The privileged program writes to the symlink, overwriting the password file. This classic TOCTOU pattern has affected sudo, sendmail, and numerous other privileged Unix programs.

Instructions

1. Recognize the TOCTOU pattern. Time-of-check-to-time-of-use vulnerabilities follow a universal structure: the program checks a condition (file exists, balance is sufficient, user is authorized, code is unused), then performs an action based on that condition (write the file, deduct the balance, grant access, apply the discount). If the condition can change between the check and the action, the action operates on a stale assumption. The fix is always the same principle: make the check and the action atomic -- either they happen as one indivisible operation, or a lock ensures nothing changes between them.

2. Use database-level atomicity for business logic. The database is the only reliable point of serialization in a multi-instance web application:

   • SELECT FOR UPDATE: Acquires a row-level lock that prevents other transactions from reading or modifying the row until the lock is released. Use for balance checks followed by balance updates: SELECT balance FROM accounts WHERE id = ? FOR UPDATE, verify balance is sufficient, then UPDATE accounts SET balance = balance - ? WHERE id = ?.
   • Serializable isolation: The strictest isolation level. Transactions execute as if they ran sequentially. Use for financial operations where phantom reads or write skew would cause incorrect results. Be aware of the performance cost -- serializable isolation increases lock contention and aborted transactions.
   • Unique constraints: For operations that must happen exactly once (coupon redemption, account creation with a unique email), enforce uniqueness at the database level with a UNIQUE constraint or UNIQUE INDEX. The database rejects the duplicate insert atomically. Do not rely on application-level checks (SELECT to see if exists, then INSERT).
   • Advisory locks: Database-specific named locks (pg_advisory_lock in PostgreSQL, GET_LOCK in MySQL) for coordinating operations that span multiple tables or require application-level logic between the check and the action.
   • Atomic updates: Use UPDATE accounts SET balance = balance - 100 WHERE id = ? AND balance >= 100 -- this combines the check and the update in a single atomic statement. If the balance is insufficient, zero rows are updated. No race window exists.

3. Implement idempotency keys for external-facing operations. Idempotency ensures that processing the same request multiple times produces the same result as processing it once:

   • The client generates a unique idempotency key (UUID) and includes it in the request.
   • The server stores the key and the result of the first processing.
   • Subsequent requests with the same key return the stored result without re-executing the operation.
   • Store idempotency keys with a UNIQUE constraint. The database prevents duplicate processing atomically.
   • Idempotency keys protect against network retries, client bugs, and deliberate duplicate submissions.

4. Use optimistic locking for low-contention updates. Add a version column to the row. Every update includes WHERE version = ? with the version the client read. If another transaction modified the row (incrementing the version), the update affects zero rows and the application knows to retry with the new version:

   • Read: SELECT id, balance, version FROM accounts WHERE id = ?
   • Update: UPDATE accounts SET balance = ?, version = version + 1 WHERE id = ? AND version = ?
   • If zero rows affected: re-read and retry (with a retry limit to prevent infinite loops)
   • Optimistic locking avoids holding database locks during user think-time and is preferred for web applications where contention is typically low.

5. Eliminate file system race conditions. File operations are particularly vulnerable because the file system namespace is shared:

   • Atomic file creation: Use O_CREAT | O_EXCL flags with open(). This creates the file only if it does not already exist, atomically. If the file exists (including as a symlink), the call fails. Do not use access() or stat() followed by open().
   • Operate on file descriptors, not paths: After opening a file, use fstat(), fchmod(), fchown() instead of stat(), chmod(), chown(). File descriptor operations act on the actual file, not the path -- an attacker cannot redirect them via symlink replacement.
   • Use mkstemp() or tmpfile(): These create temporary files with unique names atomically. Do not use tmpnam() or tempnam() followed by open() -- the name can be predicted and replaced between generation and use.
   • Use renameat2() with RENAME_NOREPLACE (Linux) for atomic file placement. Write content to a temporary file, then atomically rename it to the target path. The rename is atomic -- readers see either the old file or the new file, never a partial write.

6. Design for concurrent execution from the start. Assume every operation will be executed concurrently by multiple instances:

   • Never assume single-instance deployment. Even if there is one instance today, horizontal scaling or failover will introduce concurrency.
   • Use atomic compare-and-swap (CAS) operations for in-memory state in multi-threaded applications. CAS is the primitive that makes lock-free data structures possible.
   • Identify all "check-then-act" sequences in the codebase and evaluate whether the check and act are atomic. If they are not, add appropriate locking or use atomic operations.

Details

• How attackers exploit web application race conditions at scale: Tools like Turbo Intruder (Burp Suite extension) and custom scripts send dozens to hundreds of concurrent HTTP requests, all arriving at the server within a few milliseconds. The attacker does not need precise timing -- they flood the server with concurrent requests and statistical probability ensures that some requests hit the race window. A discount code that can be used once is used 50 times. A bank transfer of $100 from a $150 account is executed 10 times. A free trial activation creates 100 accounts. The defense must be correct under maximum concurrency, not just typical load.

• Database isolation levels and their race condition implications: READ UNCOMMITTED allows dirty reads (seeing uncommitted changes from other transactions). READ COMMITTED (the default in PostgreSQL and Oracle) prevents dirty reads but allows non-repeatable reads (a row read twice in the same transaction returns different values) and phantom reads (a range query returns different rows). REPEATABLE READ (the default in MySQL/InnoDB) prevents dirty and non-repeatable reads but allows phantom reads in the SQL standard (though InnoDB's implementation also prevents phantoms). SERIALIZABLE prevents all anomalies but increases contention. For financial transactions, READ COMMITTED with explicit SELECT FOR UPDATE locking is the most common practical approach -- it provides the necessary atomicity without the global performance cost of serializable isolation.

• The double-spend pattern in detail: User has $150. Two concurrent transfers of $100 each arrive. Thread A: SELECT balance FROM accounts WHERE id = 1 returns $150. Thread B: SELECT balance FROM accounts WHERE id = 1 returns $150 (Thread A has not yet updated). Thread A: balance >= 100, so UPDATE accounts SET balance = 50 WHERE id = 1. Thread B: balance >= 100 (it read $150), so UPDATE accounts SET balance = 50 WHERE id = 1. Both succeed. The user transferred $200 from a $150 balance. With SELECT FOR UPDATE, Thread B blocks until Thread A commits, then re-reads balance as $50 and correctly rejects the transfer.

Anti-Patterns

1. Check-then-act without holding a lock. Reading a value, making a decision based on that value, and then writing -- without ensuring the value has not changed. This is the fundamental race condition pattern. The gap between the check and the act is the vulnerability window. Use atomic operations, database locks, or optimistic locking to close the window.

2. Using READ COMMITTED isolation for financial transactions without explicit locking. READ COMMITTED does not prevent two transactions from reading the same balance and both deducting from it. Without SELECT FOR UPDATE or an atomic UPDATE ... WHERE balance >= ?, the double-spend pattern is exploitable. READ COMMITTED is safe for reads that do not influence writes, but financial transactions require explicit serialization.

3. Relying on application-level uniqueness checks without database constraints. Checking SELECT COUNT(*) FROM users WHERE email = ? and then INSERT INTO users (email, ...) VALUES (?, ...) if the count is zero. Two concurrent registrations with the same email both read count = 0 and both insert. Add a UNIQUE constraint on the email column and handle the constraint violation error -- the database enforces uniqueness atomically.

4. Non-atomic file operations with predictable temporary filenames. Using tempnam() to generate a filename, then open() to create it. An attacker predicts the filename (or creates symlinks for all likely names) and interposes between generation and creation. Use mkstemp() which generates and opens the file in a single atomic operation.

5. Assuming single-threaded execution in a multi-instance deployment. Code that uses in-memory locks (synchronized in Java, threading.Lock in Python) to prevent race conditions. These locks protect against concurrency within a single process but do nothing when the application runs as multiple instances behind a load balancer. The lock must be at the shared state level -- typically the database or a distributed lock (Redis SETNX, ZooKeeper, etcd).