redis-engineering

Redis Engineering

Ortak Doktrin

agents/shared/severity-rubric.md ve agents/shared/escalation-matrix.md default-load sayılır (agents/coordination.md §11). Bu skill'in çıktısı Critical / High / Medium / Low + kanıt formatında olmak zorunda — spekülatif Critical yasak. Sahiplik dışı bulgu ilgili agent'a delege:

• realtime-systems-reviewer — pub-sub backplane WS scale
• security-reviewer — rate limit auth integration, OWASP A04
• database-optimizer — cache invalidation Postgres-bağımlı (write-through)
• performance-profiler — memory/latency genel

Felsefe

• Cache ledger değil. Postgres source-of-truth.
• TTL ZORUNLU her cache key'inde + jitter.
• Eviction policy explicit ve workload-uyumlu.
• Atomicity Lua + pipeline. Race window'lara karşı.
• Pub-sub at-most-once. Durability gerekiyorsa Streams (scope dışı).
• Big key yasak (> 100KB).
• KEYS yasak prod; SCAN.
• HA Sentinel veya Cluster; tek-node prod SPOF.

Ne Zaman Kullanılır

• Yeni cache layer ekleniyor / cache hit rate düşük
• OOM error veya memory growth alarmı
• WS scale: 1 → N replica fanout
• Rate limit endpoint koruma kurulumu
• Slow log incelemesi (p99 latency artışı)
• Sentinel kurulumu / failover drill
• Big key / subscriber leak teşhis
• Persistence (RDB/AOF) tuning
• Cache stampede / thundering herd incident

Tooling Matrisi

Konu	Komut / araç
Genel inspect	redis-cli INFO, CLIENT LIST, CONFIG GET *
Big key	redis-cli --bigkeys, MEMORY USAGE <key>
Key scan	SCAN 0 MATCH "..." COUNT 100
Slow log	SLOWLOG GET 25
Latency	redis-cli --latency, --latency-history
Memory	INFO memory (used_memory_human, mem_fragmentation_ratio)
Persistence	INFO persistence, LASTSAVE, BGSAVE
Pub-sub	PUBSUB CHANNELS, PUBSUB NUMSUB, PUBSUB NUMPAT
Replication	INFO replication, ROLE
Sentinel	SENTINEL masters, SENTINEL sentinels mymaster

Workflow

1) Discovery — current state

# Sürüm + bellek
redis-cli INFO server | grep -E "redis_version|os|arch_bits"
redis-cli INFO memory | grep -E "used_memory_human|maxmemory_human|maxmemory_policy|mem_fragmentation_ratio"
redis-cli INFO stats | grep -E "instantaneous_ops|total_commands|keyspace_hits|keyspace_misses"

# Hit rate
HITS=$(redis-cli INFO stats | grep keyspace_hits | cut -d: -f2 | tr -d '\r')
MISS=$(redis-cli INFO stats | grep keyspace_misses | cut -d: -f2 | tr -d '\r')
echo "scale=4; $HITS/($HITS+$MISS)" | bc
# Cache: target > 0.85; çok düşükse TTL/eviction/key tasarımı sorunu

# Key sayısı + distribution
redis-cli DBSIZE
redis-cli INFO keyspace

2) Cache audit

# Eviction policy
redis-cli CONFIG GET maxmemory-policy
# noeviction = OOM yazma reddi; allkeys-lru / volatile-lru kullanım'a göre

# maxmemory
redis-cli CONFIG GET maxmemory
# 0 = sınırsız (host RAM); set zorunlu

# TTL'siz key tarama
redis-cli --scan --pattern '*' | head -100 | while read k; do
    ttl=$(redis-cli TTL "$k")
    [ "$ttl" = "-1" ] && echo "NO_TTL: $k"
done
# Çıktı çoksa: cache key'lerinde TTL yok → memory leak riski

3) Stampede mitigation tasarım

İki seçenek:

A. Lock + cache-aside:

def get_cached(key: str, loader):
    val = redis.get(key)
    if val: return val
    if redis.set(f"lock:{key}", "1", nx=True, ex=5):
        try:
            fresh = loader()
            redis.setex(key, 3600 + jitter(), fresh)
            return fresh
        finally:
            redis.delete(f"lock:{key}")
    else:
        time.sleep(0.05)
        return redis.get(key) or loader()  # fallback waiter

B. XFetch (probabilistic early refresh):

def get_cached_xfetch(key: str, loader, beta=1.0):
    val_with_delta = redis.hgetall(key)
    if val_with_delta:
        ttl = redis.ttl(key)
        delta = float(val_with_delta.get("delta", "0"))
        # Stochastic refresh: TTL azaldıkça olasılık artar
        if random.random() < math.exp(-beta * delta / ttl):
            fresh = loader()
            redis.hset(key, mapping={"val": fresh, "delta": elapsed_to_load()})
            redis.expire(key, 3600 + jitter())
            return fresh
        return val_with_delta["val"]
    fresh = loader()
    redis.hset(key, mapping={"val": fresh, "delta": elapsed_to_load()})
    redis.expire(key, 3600 + jitter())
    return fresh

4) Pub-Sub backplane (multi-replica WS)

# Backend publish (any service)
redis.publish("notifications", json.dumps({"user_id": 42, "msg": "..."}))

# WS replica (N kopya)
class WsManager:
    def __init__(self):
        self.local_clients: dict[int, set[WebSocket]] = {}

    async def listen(self):
        pubsub = self.redis.pubsub()
        await pubsub.subscribe("notifications")
        try:
            async for msg in pubsub.listen():
                if msg["type"] != "message":
                    continue
                payload = json.loads(msg["data"])
                clients = self.local_clients.get(payload["user_id"], set())
                for ws in clients:
                    await ws.send_json(payload)
        finally:
            await pubsub.unsubscribe("notifications")
            await pubsub.close()

Subscriber leak teşhis:

redis-cli CLIENT LIST | awk '$8 ~ /sub=[1-9]/ {print}'   # sub > 0 client'lar
redis-cli PUBSUB CHANNELS                                 # aktif kanal
redis-cli PUBSUB NUMSUB channel1                          # her kanaldaki subscriber count

sub=N artıyor ama replica sayısı sabit → leak. Eski PID'lere ait subscription kapanmamış. CLIENT KILL ID <id> veya restart.

5) Rate limit Lua atomic

RATE_LIMIT_LUA = """
local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

local data = redis.call('HMGET', key, 'tokens', 'last_refill')
local tokens = tonumber(data[1]) or capacity
local last_refill = tonumber(data[2]) or now

local elapsed = math.max(0, now - last_refill)
tokens = math.min(capacity, tokens + elapsed * refill_rate)

if tokens < 1 then
    redis.call('HSET', key, 'tokens', tokens, 'last_refill', now)
    redis.call('EXPIRE', key, 3600)
    return {0, tokens}
end

tokens = tokens - 1
redis.call('HSET', key, 'tokens', tokens, 'last_refill', now)
redis.call('EXPIRE', key, 3600)
return {1, tokens}
"""

sha = redis.script_load(RATE_LIMIT_LUA)

def check_rate_limit(user_id: int, capacity=60, refill_rate=1.0) -> bool:
    allowed, remaining = redis.evalsha(
        sha, 1, f"rl:{user_id}", capacity, refill_rate, time.time(),
    )
    return bool(allowed)

6) Memory audit

# Big key
redis-cli --bigkeys

# Tek key bytes
redis-cli MEMORY USAGE order:cart:42 SAMPLES 0

# Top 100 key sample (Redis 6.2+)
redis-cli --memkeys --memkeys-samples 100

# Fragmentation
redis-cli INFO memory | grep mem_fragmentation_ratio
# > 1.5 → defrag düşün
redis-cli CONFIG SET activedefrag yes
redis-cli CONFIG SET active-defrag-threshold-lower 10

7) Slow log

redis-cli CONFIG SET slowlog-log-slower-than 10000   # 10ms
redis-cli SLOWLOG GET 25 | head -50

Sık çıkanlar:

• KEYS * → SCAN ile değiştir
• HGETALL bigkey → HSCAN
• LRANGE 0 -1 → keyset-style (LPOP/RPOP batch veya ZRANGEBYSCORE)
• SMEMBERS huge_set → SSCAN

8) Persistence

redis-cli INFO persistence
# rdb_last_save_time, rdb_changes_since_last_save
# aof_enabled, aof_last_rewrite_time_sec

# Cache-only — persistence kapat
redis-cli CONFIG SET save ""
redis-cli CONFIG SET appendonly no

# Source-of-truth — AOF everysec
redis-cli CONFIG SET appendonly yes
redis-cli CONFIG SET appendfsync everysec

9) Sentinel HA

# Sentinel info
redis-cli -p 26379 SENTINEL masters
redis-cli -p 26379 SENTINEL sentinels mymaster
redis-cli -p 26379 SENTINEL get-master-addr-by-name mymaster

# Manual failover drill (staging)
redis-cli -p 26379 SENTINEL failover mymaster

# Min replicas
redis-cli CONFIG SET min-replicas-to-write 1
redis-cli CONFIG SET min-replicas-max-lag 10

App tarafı: sentinel-aware client zorunlu (örn. redis-py Sentinel, ioredis sentinel mode). Direct master IP yazılırsa failover sonrası bağlantı yenilenmez.

10) Bulgu raporu

# Redis Findings: <cluster/service>

## Critical
- [ ] `maxmemory` set değil + persistence on → OOM kernel kill, RDB corrupt
      riski — `redis.conf` veya `CONFIG SET maxmemory 4gb`

## High
- [ ] Eviction policy `noeviction` cache için → yazma reddi —
      `CONFIG SET maxmemory-policy allkeys-lru`
- [ ] 1.4M key TTL'siz (cache) — `SCAN` ile sample 100, %78 TTL=-1 →
      memory leak; `setex` zorunlu hale getir, mevcut'lara backfill TTL

## Medium
- [ ] Slow log: 12 `KEYS *` query saatte; p99 320ms — `SCAN` migrasyonu
- [ ] `mem_fragmentation_ratio` 1.83 — `activedefrag` enable

## Low
- [ ] Sentinel quorum 3 (3 sentinel) — best practice 5 (split-brain
      düşük olasılık)

Checklist

• maxmemory set + workload'a uygun
• maxmemory-policy cache için allkeys-lru/lfu veya karma için volatile-lru/lfu
• Tüm cache key'lerinde TTL var; TTL jitter eklenmiş
• Cache stampede mitigation (lock + refresh-ahead veya XFetch)
• Pub-sub subscriber close edilmiş (finally bloğu)
• Rate limit Lua atomic (GETSET değil)
• Big key teşhisi yapılmış (--bigkeys); > 100KB key yok
• Slow log threshold 10ms; KEYS/HGETALL bigkey yok
• mem_fragmentation_ratio < 1.5
• Persistence stratejisi cache vs source-of-truth ayrımı net
• HA: Sentinel (3+ nodes, quorum 2) veya Cluster; tek-node prod yok
• App sentinel-aware client kullanıyor (direct master IP değil)
• Subscriber/connection leak teşhisi (CLIENT LIST | sub=)

Antipattern

• TTL'siz cache key — memory leak garantisi.
• noeviction cache için — yazma reddi prod incident.
• volatile-lru + TTL'siz key karışımı — TTL'siz'ler büyür sınırsız.
• KEYS * prod'da — O(N) blocking.
• Big key > 100KB — blocking command latency.
• Pub-sub durable beklentisi — Streams kullan.
• Rate limit pipeline / GETSET — race window.
• TTL jitter yok — synchronized expiry thundering herd.
• mem_fragmentation_ratio ignored.
• Tek-node prod.
• Direct master IP app config'inde — failover'da kopuk.
• Cache key cookie/query param hash — invalidation imkânsız.
• Source-of-truth Redis, Postgres yok — data loss kabul edilemez bağlamda.
• AOF appendfsync always cache için — write latency artar, kazanım yok.
• Persistence kapalı + Redis sequence/counter source-of-truth — restart data loss.

Cross-Link

• rules/redis.md — discipline rule.
• rules/websocket.md — pub-sub backplane fanout protocol.
• rules/security.md — rate limit OWASP A04 cross-link.
• skills/websocket-realtime-systems/SKILL.md — WS scale + Redis backplane.
• skills/postgres-performance/SKILL.md — cache-aside + write-through DB invalidation pattern.
• agents/database-optimizer.md — Postgres tarafı (Redis cache invalidation Postgres write trigger ile).
• agents/realtime-systems-reviewer.md — pub-sub backplane sahiplik.
• commands/redis-review.md — slash command entrypoint.
• MCP: redis (read-only) — INFO, CLIENT LIST, SLOWLOG, MEMORY USAGE.