Pandera Validation

Audience: Data engineers validating pandas DataFrames.

Goal: Provide pandera patterns for schema validation and type checking.

Scripts

Execute schema functions from scripts/schemas.py:

from scripts.schemas import (
    create_user_schema,
    create_nullable_schema,
    create_date_range_schema,
    UserSchema,
    validate_with_errors,
    infer_and_export_schema
)

Usage Examples

Basic Schema Validation

from scripts.schemas import create_user_schema

schema = create_user_schema()
validated_df = schema.validate(df)

Collect All Errors

from scripts.schemas import create_user_schema, validate_with_errors

schema = create_user_schema()
validated_df, errors = validate_with_errors(df, schema)

if errors:
    for err in errors:
        print(f"{err['column']}: {err['check']} - {err['failure_case']}")

Class-Based Schema

from scripts.schemas import UserSchema

# Validate with type hints
UserSchema.validate(df)

# Use as function type hint
def process_users(df: pa.typing.DataFrame[UserSchema]) -> pd.DataFrame:
    return df.query("status == 'active'")

Infer Schema from DataFrame

from scripts.schemas import infer_and_export_schema

schema_export = infer_and_export_schema(df)
print(schema_export['python_code'])  # Python schema definition
print(schema_export['yaml'])         # YAML schema

Built-in Checks Reference

Check Type	Example	Description
Numeric	Check.gt(0), Check.in_range(0, 100)	Comparisons
String	Check.str_matches(r'pattern')	Regex match
Set membership	Check.isin(['A', 'B'])	Allowed values
Uniqueness	unique=True on Column	No duplicates
Nullable	nullable=True on Column	Allow nulls

Decorator-Based Validation

import pandera as pa

@pa.check_output(schema)
def load_data(path: str) -> pd.DataFrame:
    return pd.read_csv(path)

@pa.check_input(schema, "df")
def process_data(df: pd.DataFrame) -> pd.DataFrame:
    return df.assign(processed=True)

@pa.check_io(df=input_schema, out=output_schema)
def transform_data(df: pd.DataFrame) -> pd.DataFrame:
    return df.transform(...)

When to Use Pandera

Use Case	Pandera	Alternative
DataFrame validation	✓	-
Type hints for DataFrames	✓	-
ETL pipeline checks	✓	Great Expectations
Record-level validation	-	Pydantic

Dependencies

pandera>=0.18
pandas