Kafka plugins

Build production-ready data pipelines with Apache Airflow and dbt, manage scalable data warehouses, and implement vector search and RAG systems using embedding models and vector databases.

Delegate complex data engineering, ML, and AI workflows to specialized sub-agents that design scalable pipelines, build and optimize models, architect LLM systems, tune databases for performance, and deploy production infrastructure across clouds.

Implement Domain-Driven Design with tactical patterns (aggregates, entities, value objects), CQRS, event sourcing, and saga orchestration. Design bounded contexts, map integration contracts, and build event stores with read models for scalable, event-driven systems.

Create, validate, and debug Redpanda Connect pipeline configurations and Bloblang transformation scripts from natural language descriptions, with component discovery and error repair for streaming data workflows.

Develop and manage end-to-end Databricks workflows: data engineering with Spark, Delta, Iceberg, and streaming; build and deploy ML models and GenAI agents; create dashboards, apps, and CI/CD bundles; query Unity Catalog and manage infrastructure via SDK/CLI operations.

Interactively select architecture paradigms for software systems via scenario matching, compare trade-offs for team size and complexity, generate ADRs, and receive tailored implementation plans with steps, deliverables, risks, and mitigations for client-server, CQRS, event-driven, hexagonal, microservices, and more.

Delegate data engineering to AI agents that design ETL/ELT pipelines with orchestration via Airflow, Prefect, or Dagster; engineer data warehouses with star schemas, partitioning, and query optimization; and build streaming systems using Kafka, Flink, or Spark Streaming for real-time processing and monitoring.

Connect to MongoDB databases to explore data, manage collections, optimize queries and indexes, generate queries from natural language, implement Atlas Search (including vector and hybrid search), configure stream processing workflows, and tune client connection settings across drivers.

Build, query, and manage Neo4j graph databases across the full stack: Cypher query generation and optimization, driver integration (Python, Java, Go, .NET, JavaScript), GraphRAG and vector search pipelines, data ingestion and import, Aura cloud provisioning, Kafka streaming, GraphQL API generation, and performance tuning.

Design and implement data foundations for financial systems: reference and market data management, integration architectures with REST APIs, WebSockets, gRPC, Kafka, FIX protocol, ISO 20022, batch feeds; plus data quality programs with validation, lineage, governance for regulatory compliance like BCBS 239 and MiFID II.

Design scalable distributed systems architectures for APIs, data pipelines, ML/RAG, edge/CDN, chaos engineering, and observability; review for security, resilience, performance, and quality attributes; simulate mock system design interviews with feedback.

Author, convert between, validate, refine, and audit specifications using EARS, Gherkin, Kiro, and user story formats. Generate ADRs, implementation plans, and run GitHub SpecKit's 5-phase workflow from constitution to implementation for spec-driven development.

Delegate SDLC workflows to specialist AI agents that architect cloud-native systems, design databases, conduct deep web research, optimize performance and observability, distill repo knowledge, and build production agents via orchestrated pipelines.

Audits Kafka clusters for security misconfigurations, topic compliance, connector health, consumer lag, schema compatibility, and performance anti-patterns via the Lenses MCP server, with remediation recommendations.

Design contract-first APIs via OpenAPI 3.1, AsyncAPI 3.0, and Protocol Buffers from natural language requirements. Author formal specifications for systems, algorithms, and behaviors using TLA+, SysML, UML diagrams, state machines in Mermaid/PlantUML, and Use Case 2.0 templates for verification and modeling workflows.

Review Spring Boot codebases using skill detection and parallel agents for DDD patterns, security, testing, observability, and Modulith modules. Verify upgrades to Spring Boot 4 with multi-phase analysis producing severity-categorized migration checklists and remediation steps. Implement data layers, REST APIs, and configurations via guided skills.

Write and review event-driven Go microservices using the xgodev/boost framework: wire Kafka/NATS/Pub-Sub consumers, configure factories for databases, caches, and cloud SDKs, compose middleware chains, and manage multi-listener graceful shutdown.

Design optimized message queue systems for Kafka, RabbitMQ, SQS using queue theory for throughput and latency. Implement async processing with workers, retries, backoff; manage backpressure via circuit breakers, rate limiting; architect distributed systems with consensus like Raft, CAP theorem, fault tolerance.

Process and transform data using jq, SQL, or pandas; design ETL/ELT pipelines for batch or streaming; perform time series forecasting, anomaly detection, and analytics; architect streaming systems with Kafka; generate insights and visualizations via natural language commands and specialist agents.

Orchestrate the full SDLC from a single CTO-style prompt: agents produce architecture docs, decompose work into tasks, implement with TDD, run domain-specific compliance reviews (HIPAA, PCI, FDA, SOC2, fintech, AI bias, etc.), and gate deploys on security sign-off. You approve two decisions per feature.

Reverse engineer unfamiliar codebases by running unwind:start to orchestrate layer-by-layer analysis from database schemas and domain models to services, APIs, frontend routes, messaging, and tests, generating structured Markdown docs in docs/unwind/layers/, detecting gaps in coverage, and producing architecture summaries plus rebuild plans.

Run first-principles study sessions on PDFs, URLs, GitHub repos, and YouTube videos; generate structured research summaries and Study Vault notes; write technical blog posts with Notion integration; set up Docker Compose lab environments for hands-on practice; and automate daily spaced-repetition quizzes via Slack and GitHub Actions.