majestic-data

Generate or execute data migration scripts with safety checks.

Generate comprehensive data profile for a file or table.

Transform data files between formats or apply transformations.

Run data validation suite against a dataset.

Generate test suites for ETL pipelines using fixture generation and testing patterns.

Design and execute historical data backfill strategies with progress tracking.

Plan and coordinate multi-step ETL pipelines with dependency management.

Detect anomalies in data using statistical and ML methods. Z-score, IQR, Isolation Forest, and time-series anomalies.

Handle messy CSVs with encoding detection, delimiter inference, and malformed row recovery.

Generate data profiles with column stats, correlations, and missing patterns for DataFrames. Use for EDA and data discovery.

Quality dimensions, scorecards, distribution monitoring, and freshness checks. Use for data validation pipelines and quality gates.

Data validation patterns and pipeline helpers. Custom validation functions, schema evolution, and test assertions.

dbt (data build tool) patterns for model organization, incremental strategies, and testing.

Core ETL reliability patterns including idempotency, checkpointing, error handling, chunking, retry logic, and logging.

Incremental data loading patterns including backfill strategies, CDC, timestamp-based loads, and pipeline orchestration.

Production ETL patterns orchestrator. Routes to core reliability patterns and incremental load strategies.

Data validation using Great Expectations. Expectation suites, checkpoints, and data docs for pipeline monitoring.

DataFrame manipulation with chunked processing, memory optimization, and vectorized operations.

DataFrame schema validation using pandera. Schema definitions, column checks, and decorator-based validation.

Columnar file patterns including partitioning, predicate pushdown, and schema evolution.

Record-level data validation using Pydantic models. Field validators, model validators, and batch validation patterns.

Infer schema from sample data files (CSV, JSON, Parquet) and generate type definitions.

Analyze data source characteristics including update frequency, volume patterns, and schema stability.

Advanced SQL patterns including window functions, CTEs, recursive queries, and optimization techniques.

Generate synthetic test data with edge cases for ETL pipeline testing.

Pytest templates and patterns for ETL pipeline testing - unit, integration, data quality.