Distributed Systems Patterns

Overview

Don't distribute unless you must. When you must, choose the lowest escalation level that meets requirements. Each level adds complexity, failure modes, and operational burden — only escalate when you've outgrown the current level.

Escalation Decision

Single node handles the load?
  YES → Level 0 (don't distribute)
  NO  → Do you need shared state across nodes?
          NO  → Level 1 (stateless multi-node)
          YES → Is eventual consistency acceptable?
                  YES → Is it just presence/registry?
                          YES → Level 2 (lightweight)
                          NO  → Level 4 (CRDTs)
                  NO  → Level 3 (Raft consensus)
                         Cluster > 50 nodes? → Level 5

See escalation-ladder.md for full details on each level with code examples and migration triggers.

Quick Decision

Need exactly-one process across cluster?  → leader-election.md
Need multi-service transactions?          → distributed-transactions.md
Need to partition data across nodes?      → consistent-hashing.md
Need audit trail / event replay?          → event-sourcing.md
Need strong consistency?                  → consensus.md (Raft)
Need eventual consistency?                → consensus.md (CRDTs)
Need cluster membership / discovery?      → clustering.md, gossip-protocols.md
Need to understand failure modes?         → failure-modes.md
Preparing for production deployment?      → production-checklist.md

Common Mistakes

• Distributing too early: A single BEAM node handles millions of processes. Start at Level 0.
• Choosing Raft when CRDTs suffice: If eventual consistency is acceptable, CRDTs avoid all consensus complexity.
• Ignoring asymmetric partitions: Real networks produce one-way failures that violate Raft assumptions.
• Single Global Process: GenServer-as-cache becomes a bottleneck and SPOF on a cluster.
• Trusting CP/AP labels: Verify actual guarantees, don't trust vendor marketing.

Reference Files

Read the file that matches your current problem:

• leader-election.md — When: Need exactly-one process across cluster. :global, :pg, Horde, Oban cron, singleton patterns, netsplit behavior for each approach
• distributed-transactions.md — When: Multiple services must coordinate atomically. Saga (choreography + orchestration), Oban workflows, compensating transactions, why not 2PC
• gossip-protocols.md — When: Need efficient cluster-wide information spread. SWIM failure detection, epidemic broadcast, delta-CRDTs, HyParView/Partisan
• consistent-hashing.md — When: Partitioning data or load across nodes. Ring hashing, jump consistent hash, virtual nodes, sharded ETS, Broadway partitioning
• event-sourcing.md — When: Need audit trail or event replay. Event sourcing, CQRS, Commanded patterns, projections, when to use vs CRUD
• escalation-ladder.md — When: Deciding whether/how to distribute. Full Distribution Escalation Ladder (Levels 0-5) with code examples
• consensus.md — When: Need strong or eventual consistency guarantees. Raft via :ra, CRDTs via delta_crdt, Multi-Raft, Vector Clocks, Paxos
• clustering.md — When: Setting up node discovery and communication. Distributed Erlang, Partisan, libcluster configs, PubSub, debugging patterns
• failure-modes.md — When: Debugging distributed bugs or designing for resilience. Split-brain, fencing tokens, clock drift, gray failures, metastable failures, limplocks, concurrency bugs, blast radius
• cap-tradeoffs.md — When: Evaluating consistency vs availability tradeoffs. CAP theorem critique, fundamental limits, consistency models
• production-checklist.md — When: Preparing distributed system for deployment. 20-item deployment checklist
• resilience-principles.md — When: Designing fault-tolerant architecture. Architectural principles from Hebert, Brooker, Luu, Kleppmann, Cook et al.

Commands

• /distributed-review — Deep analysis of distributed architecture and failure modes
• /feature <desc> — Guided implementation with distribution-aware design

Related Skills

• elixir-patterns: GenServer, Supervisor, OTP patterns, overload management
• production-quality: Monitoring, observability, error handling
• phoenix-liveview: Phoenix.PubSub, Phoenix.Tracker

Use the distributed-systems-expert agent for deep analysis of distributed architectures, consensus algorithm selection, and distributed bug investigation.