Message queue and job processing -- Kafka, RabbitMQ, SQS, BullMQ, Celery, Sidekiq.
From godmodenpx claudepluginhub arbazkhan971/godmodeThis skill uses the workspace's default tool permissions.
Designs and optimizes AI agent action spaces, tool definitions, observation formats, error recovery, and context for higher task completion rates.
Enables AI agents to execute x402 payments with per-task budgets, spending controls, and non-custodial wallets via MCP tools. Use when agents pay for APIs, services, or other agents.
Compares coding agents like Claude Code and Aider on custom YAML-defined codebase tasks using git worktrees, measuring pass rate, cost, time, and consistency.
/godmode:queue, "add background jobs", "set up queue"grep -r "bullmq\|celery\|sidekiq\|kafkajs\|sqs" \
package.json requirements.txt 2>/dev/null
Use case: job processing | event streaming | pub/sub
Volume: <msg/sec>, Payload: <avg size>
Ordering: strict FIFO | partition | best-effort
Delivery: exactly-once | at-least-once | at-most-once
Latency: <100ms | <1s | <30s
Existing infra: Redis | PostgreSQL | AWS | none
SQS: AWS-native, managed, unlimited throughput
BullMQ: Node.js+Redis, 10-50K/sec, great dashboard
Celery: Python+Redis/RabbitMQ, complex workflows
Kafka: millions/sec, partition-ordered, replayable
RabbitMQ: complex routing, 10-50K/sec
Redis Streams: lightweight, 100K+/sec
PG SKIP LOCKED: no new infra, <1K jobs/sec
IF AWS simple: SQS. IF Node.js+Redis: BullMQ. IF event streaming: Kafka. IF low volume+PG: SKIP LOCKED.
Producers -> Broker
-> [high-priority] -> Worker Pool A (concurrency 10)
-> [default] -> Worker Pool B (concurrency 20)
-> [bulk] -> Worker Pool C (concurrency 5)
-> [dead-letter] -> DLQ Processor
Retry: 0s -> 1s -> 4s -> 16s -> 60s (cap) -> DLQ
Formula: min(base * 2^attempt + jitter, max_delay)
Retryable: network timeout, 5xx, DB connection, 429
Non-retryable: 4xx, auth, deserialization, biz logic
DLQ: <original>-dlq, retention 30 days, alert >100
Options: replay | replay with fix | skip | escalate
Idempotency: check Redis key, acquire lock (NX+TTL), process, store result (TTL 24h), release lock.
P0 critical: password reset, payment (SLA <10s)
P1 high: welcome email, order confirm (SLA <60s)
P2 normal: notifications, image proc (SLA <5m)
P3 low: reports, exports (SLA <1h)
P4 background: cleanup, analytics (SLA <24h)
Rate limit: token bucket or BullMQ limiter
{ max: 100, duration: 60000 }.
Min workers: 2, Max: 20, Scale on depth > 100
Concurrency per worker: 10 (I/O-bound)
Backpressure:
Depth > 1K -> WARN + alert
Depth > 10K -> SCALE workers
Depth > 100K -> SHED low-priority
Worker memory > 80% -> PAUSE accepting
Downstream 5xx > 10% -> CIRCUIT BREAK
Graceful shutdown: SIGTERM -> pause -> finish -> exit.
Daily digest: 0 9 * * * (default, 5m timeout)
Cleanup: 0 */6 * * * (bg, 30m timeout)
Weekly reports: 0 2 * * 1 (bulk, 2h timeout)
Distributed lock (one scheduler), all UTC, alert if job misses window.
Queue depth waiting (<1K), active (<50), DLQ (<100)
Processing rate (>10/s), Success rate (>95%)
Avg time (<10s), P95 (<30s)
Worker count (>2), Retry rate (<10%)
Append .godmode/queue-results.tsv:
timestamp technology queues worker_pools retry dlq idempotency status
KEEP if: depth stable AND error rate < 1%
AND P95 < target AND DLQ not growing.
DISCARD if: depth growing OR error spike
OR DLQ growing.
STOP when ALL of:
- All >500ms tasks queued
- Retry + backoff configured
- DLQ on all queues
- Idempotent handlers
- Graceful shutdown
- Error rate < 1%
On failure: git reset --hard HEAD~1. Never pause.
| Failure | Action |
|---|---|
| Jobs stuck active | Check logs, verify timeout |
| DLQ growing | Inspect patterns, fix root cause |
| Memory exhaustion | Limit concurrency, check leaks |
| Duplicates | Verify idempotency key, add dedup |
| Backlog growing | Scale workers, add backpressure |