dual-write

Dual Write Migration Pattern

Dual write keeps two data stores in sync by writing to both the old and new system on every mutation. This enables gradual migration with rollback safety.

When to Use This Skill

Use this skill when...	Use shadow-mode instead when...
Migrating between databases or schemas	Validating read-path behavior under real traffic
Switching storage backends (SQL to NoSQL, etc.)	Testing a new service without writing to it
Need both systems to stay authoritative during transition	Only need to compare responses, not persist data
Planning zero-downtime data migrations	Mirroring traffic to a staging environment
Reviewing code that writes to multiple data stores	Evaluating performance of a replacement system

Core Concepts

Migration Phases

Phase	Primary reads	Primary writes	Secondary writes	Duration
1. Prepare	Old	Old	None	Setup
2. Dual write	Old	Old + New	New (async or sync)	Migration window
3. Backfill	Old	Old + New	New	Until parity
4. Shadow read	Old + New (compare)	Old + New	New	Validation
5. Cutover	New	New	Old (optional)	Transition
6. Cleanup	New	New	None	Final

Write Strategies

Strategy	Consistency	Latency impact	Failure mode
Synchronous	Strong	Higher (2x write)	Fail if either store fails
Async secondary	Eventual	Minimal	Secondary may lag
Outbox pattern	Eventual	Minimal	Requires message broker
Change data capture	Eventual	None (DB-level)	Requires CDC infrastructure

Implementation Architecture

Synchronous Dual Write

Client Request
    │
    ▼
┌─────────────┐
│  Application │
│    Layer     │
└──────┬──────┘
       │ write(data)
       ▼
┌─────────────┐
│  Dual Write  │
│  Adapter     │
├──────┬──────┤
│      │      │
▼      │      ▼
Old DB │   New DB
       │
   Compare on
   read (optional)

Key Components

Component	Responsibility
Write adapter	Routes writes to both stores, handles failures
Read comparator	Reads from both, logs discrepancies, returns primary
Backfill job	Copies historical data from old to new store
Reconciliation	Detects and resolves drift between stores
Feature flags	Controls which phase is active per entity/tenant

Implementation Patterns

Write Adapter Pattern

The write adapter wraps both stores behind a single interface:

1. Accept the write request
2. Write to the primary (old) store first
3. Write to the secondary (new) store
4. If secondary fails: log the failure, enqueue for retry, do not fail the request
5. Return the primary store's result to the caller

Read Comparison Pattern

During the shadow read phase:

1. Read from the primary (old) store — this is the authoritative response
2. Read from the secondary (new) store asynchronously
3. Compare results field by field
4. Log discrepancies with context (entity ID, field, old value, new value)
5. Return the primary store's result
6. Track comparison metrics (match rate, common divergence fields)

Backfill Strategy

1. Snapshot the old store at a known point in time
2. Begin dual writes for all new mutations
3. Copy historical records in batches (oldest first or by priority)
4. Track backfill progress per entity type
5. Reconcile records modified during backfill (dual write wins over backfill)

Failure Handling

Failure scenario	Response	Recovery
Secondary write fails	Log, continue, enqueue retry	Async retry with backoff
Primary write fails	Fail the request (do not write to secondary)	Standard error handling
Both fail	Fail the request	Standard error handling
Secondary write timeout	Log, continue	Async verification and repair
Inconsistency detected	Log with full context	Manual or automated reconciliation

Consistency Guarantees

• Primary store is always the source of truth until cutover
• Secondary store may lag during async dual write
• Reconciliation jobs detect and repair drift
• Cutover only happens when match rate reaches threshold (e.g., 99.9%)

Cutover Decision Criteria

Metric	Threshold	How to measure
Read comparison match rate	> 99.9%	Shadow read comparison logs
Backfill completion	100%	Backfill progress tracker
Secondary write success rate	> 99.95%	Write adapter metrics
P99 latency impact	< 20% increase	Application metrics
Reconciliation gap	0 unresolved	Reconciliation job output

Common Pitfalls

Pitfall	Mitigation
Ordering issues between stores	Use idempotent writes, include version/timestamp
Transaction boundaries differ	Design writes to be independently valid
Schema mismatch between stores	Map fields explicitly, handle nullability differences
Backfill conflicts with live writes	Live dual writes take precedence over backfill
Performance degradation	Start with async secondary writes
Partial failures leave inconsistency	Reconciliation job as safety net
Forgetting to dual-write in all code paths	Centralize through write adapter, audit call sites

Rollback Plan

Phase	Rollback action	Data impact
Dual write	Stop writing to new store	No data loss
Shadow read	Stop comparing reads	No data loss
Cutover (reads)	Switch reads back to old	No data loss if still dual-writing
Cutover (writes)	Reverse write order	May need reconciliation
Cleanup	Cannot rollback	Old store decommissioned

Monitoring Checklist

• Write success rate per store
• Write latency per store (P50, P95, P99)
• Read comparison match rate
• Backfill progress percentage
• Reconciliation queue depth
• Error rate by failure type
• Feature flag state per tenant/entity

Agentic Optimizations

Context	Approach
Code review	Check that all write paths go through the dual-write adapter
Architecture review	Verify failure handling, rollback plan, and cutover criteria
Implementation	Start with write adapter + async secondary, add comparison later
Testing	Simulate secondary failures, verify primary is unaffected

Quick Reference

Term	Definition
Primary store	The authoritative data store (old system during migration)
Secondary store	The new data store being migrated to
Backfill	Copying historical data from primary to secondary
Reconciliation	Detecting and repairing differences between stores
Cutover	Switching the primary designation from old to new
Match rate	Percentage of shadow reads that return identical results
Write adapter	Abstraction layer that routes writes to both stores