everything-claude-code

Atlantis Framework - AI Assistant Repository

AI assistant framework for Atlantis - OOP-based composable ETL framework with Blackboard pattern, Pydantic metadata, and Pandera data quality.

Specialized agents for building reusable Atlantis components (SCD2Reader, SCD1Writer, etc.), Azure Synapse pipelines, Pydantic metadata models, and Pandera schema validation.

---

Atlantis Framework Overview

The Atlantis framework is an object-oriented approach to ETL development:

• Blackboard Pattern: Components communicate via shared state, no direct dependencies
• Composable Building Blocks: SCD2Reader, CustomerProcessor, SCD1Writer as reusable classes
• Pydantic Metadata: Type-safe TableMetadata and ProcessMetadata models
• Pandera Validation: Schema-based data quality checks
• Azure Synapse: Dedicated SQL pools, Spark pools, pipelines
• Monorepo: Core Python package, Azure Functions, IaC, Synapse artifacts

Example Atlantis Pipeline

from atlantis import AtlantisPipeline, Blackboard
from atlantis.components import SCD2Reader, CustomerProcessor, SCD1Writer

# Components don't know about each other - only the blackboard
pipeline = AtlantisPipeline(
    config=process_metadata,  # Pydantic ProcessMetadata model
    components=[
        SCD2Reader(source_metadata),
        CustomerProcessor(transformation_rules),
        SCD1Writer(target_metadata)
    ]
)

result = pipeline.execute()  # Components execute via blackboard

---

What's New: Data Engineering Focus 🚀

This repository has been transformed from a general software development framework to a specialized data engineering AI framework. See DATA_ENGINEERING_TRANSFORMATION.md for full details.

12 Specialized Agents Across 3 Tiers

Tier 1 (Must-Have): Developer, Unit Test, Code Review, Refactor & Cleaner, Schema & Contract
Tier 2 (High-Leverage): Data Quality, SQL Optimization, Dependency Impact, Documentation
Tier 3 (Advanced): Architecture Guard, Security Scan, Metadata

See DATA_ENGINEERING_AGENTS.md for complete agent documentation.

---

Getting Started

Quick Start

1. Install Prerequisites

   • Python 3.8 or higher
   • VS Code with GitHub Copilot extension
   • Git
   • Connection to your data warehouse (Snowflake, BigQuery, etc.)

2. Clone and Install

git clone https://github.com/yourusername/ai-data-engineering.git
cd ai-data-engineering

# Install dependencies
pip install -r requirements.txt

# For development (includes pytest, black, flake8)
pip install -r requirements-dev.txt

3. Configure Package Manager

python scripts/setup_package_manager.py

4. Start Using Agents

# Write a data pipeline
@developer.md Write a PySpark pipeline to aggregate customer orders

# Create tests
@unit-test.md Create pytest tests for the aggregation function

# Review code
@code-reviewer.md Review this pipeline for idempotency and partitioning

Full Setup Guide

See QUICKSTART.md for detailed setup instructions including:

• Data warehouse connection setup
• pytest configuration
• Great Expectations setup
• dbt integration

---

Cross-Platform Support

This framework fully supports Windows, macOS, and Linux. All hooks and scripts are written in Python for maximum compatibility.

Package Manager Detection

The system automatically detects your preferred Python package manager (pip, poetry, pipenv, or uv) with the following priority:

1. Environment variable: COPILOT_PACKAGE_MANAGER
2. Project config: .github-copilot/package-manager.json
3. pyproject.toml: Tool sections for poetry/pipenv/uv
4. Lock file: Detection from requirements.txt, poetry.lock, Pipfile.lock, or uv.lock
5. Global config: ~/.github-copilot/package-manager.json
6. Fallback: First available package manager

To set your preferred package manager:

# Via environment variable
export COPILOT_PACKAGE_MANAGER=poetry

# Via setup script - global
python scripts/setup_package_manager.py --global poetry

# Via setup script - project
python scripts/setup_package_manager.py --project poetry

# Detect current setting
python scripts/setup_package_manager.py --detect

Or use the /setup-pm command in VS Code.

---

What's Inside

This repo is a VS Code GitHub Copilot configuration - use it directly or copy components as needed.