Set Up Local Kafka Dev Environment

Spin up a complete local Kafka development environment with one command.

What This Command Does

1. Docker Compose Selection: Choose Kafka or Redpanda
2. Service Configuration: Kafka + Schema Registry + UI + Monitoring
3. Environment Setup: Generate docker-compose.yml
4. Start Services: docker-compose up -d
5. Verification: Test cluster and provide connection details

Two Options Available

Option 1: Apache Kafka (KRaft Mode)

Services:

• ✅ Kafka broker (KRaft mode, no ZooKeeper)
• ✅ Schema Registry (Avro schemas)
• ✅ Kafka UI (web interface, port 8080)
• ✅ Prometheus (metrics, port 9090)
• ✅ Grafana (dashboards, port 3000)

Use When: Testing Apache Kafka specifically, need Schema Registry

Option 2: Redpanda (3-Node Cluster)

Services:

• ✅ Redpanda (3 brokers, Kafka-compatible)
• ✅ Redpanda Console (web UI, port 8080)
• ✅ Prometheus (metrics, port 9090)
• ✅ Grafana (dashboards, port 3000)

Use When: Testing high-performance alternative, need multi-broker cluster locally

Example Usage

# Start dev environment setup
/sw-kafka:dev-env

# I'll ask:
# 1. Which stack? (Kafka or Redpanda)
# 2. Where to create files? (current directory or specify path)
# 3. Custom ports? (use defaults or customize)

# Then I'll:
# - Generate docker-compose.yml
# - Start all services
# - Wait for health checks
# - Provide connection details
# - Open Kafka UI in browser

What Gets Created

Directory Structure:

./kafka-dev/
├── docker-compose.yml       # Main compose file
├── .env                      # Environment variables
├── data/                     # Persistent volumes
│   ├── kafka/
│   ├── prometheus/
│   └── grafana/
└── config/
    ├── prometheus.yml       # Prometheus config
    └── grafana/             # Dashboard provisioning

Services Running:

• Kafka: localhost:9092 (plaintext) or localhost:9093 (SASL_SSL)
• Schema Registry: localhost:8081
• Kafka UI: http://localhost:8080
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)

Connection Examples

After setup, connect with:

Producer (Node.js):

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();
await producer.connect();
await producer.send({
  topic: 'test-topic',
  messages: [{ value: 'Hello Kafka!' }]
});

Consumer (Python):

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    'test-topic',
    bootstrap_servers=['localhost:9092'],
    group_id='my-group',
    auto_offset_reset='earliest'
)

for message in consumer:
    print(f"Received: {message.value}")

kcat (CLI):

# Produce message
echo "Hello Kafka" | kcat -P -b localhost:9092 -t test-topic

# Consume messages
kcat -C -b localhost:9092 -t test-topic -o beginning

Sample Producer/Consumer

I'll also create sample code templates:

• producer-nodejs.js - Production-ready Node.js producer
• consumer-nodejs.js - Production-ready Node.js consumer
• producer-python.py - Python producer with error handling
• consumer-python.py - Python consumer with DLQ

Prerequisites

• Docker 20+ installed
• Docker Compose v2+
• 4GB+ free RAM (for Redpanda 3-node cluster)
• Ports available: 8080, 8081, 9090, 9092, 9093, 3000

Post-Setup

After environment starts, I'll:

1. ✅ Open Kafka UI in browser (http://localhost:8080)
2. ✅ Create a test topic via UI
3. ✅ Show producer/consumer examples
4. ✅ Provide kcat commands for testing
5. ✅ Show Grafana dashboards (http://localhost:3000)

Useful Commands

# Start environment
docker-compose up -d

# Stop environment
docker-compose down

# Stop and remove data
docker-compose down -v

# View logs
docker-compose logs -f kafka

# Restart Kafka only
docker-compose restart kafka

# Check health
docker-compose ps

---

Skills Activated: kafka-cli-tools Docker Compose Location: plugins/specweave-kafka/docker/ Sample Code: plugins/specweave-kafka/docker/templates/