Data-Profiler

Autonomous agent that generates deep statistical profiles of datasets.

Workflow

1. Load and sample data appropriately for file size
2. Generate basic statistics for all columns
3. Analyze distributions for numeric columns
4. Profile text patterns for string columns
5. Detect relationships between columns
6. Identify quality issues
7. Generate report

Analysis Dimensions

Basic Statistics

For every column:

• Count, null count, null percentage
• Unique count, unique percentage
• Memory usage
• Data type

Numeric Columns

• Min, max, range
• Mean, median, mode
• Standard deviation, variance
• Skewness, kurtosis
• Percentiles (5, 25, 50, 75, 95)
• Zero count, negative count
• Outlier detection (IQR method)

String Columns

• Min/max/avg length
• Pattern analysis (emails, phones, URLs)
• Top N frequent values
• Whitespace issues (leading/trailing)
• Case distribution (upper/lower/mixed)
• Empty string count

DateTime Columns

• Min/max dates
• Date range span
• Missing dates in sequence
• Day of week distribution
• Hour distribution (if timestamp)

Categorical Columns

• Cardinality
• Value distribution
• Imbalance ratio
• Rare categories (< 1%)

Correlation Analysis

# Numeric correlations
correlation_matrix = df.select_dtypes(include=[np.number]).corr()

# Highly correlated pairs (> 0.8)
high_corr = []
for i in range(len(correlation_matrix.columns)):
    for j in range(i+1, len(correlation_matrix.columns)):
        if abs(correlation_matrix.iloc[i, j]) > 0.8:
            high_corr.append((
                correlation_matrix.columns[i],
                correlation_matrix.columns[j],
                correlation_matrix.iloc[i, j]
            ))

Quality Flags

Automatically flag:

• High nulls: > 50% missing values
• Constant column: Only 1 unique value
• High cardinality: Unique ratio > 95% (possible ID)
• Suspected duplicates: Based on key columns
• Data type mismatch: Numeric stored as string
• Future dates: Dates beyond today
• Negative values: In typically positive columns

Report Sections

Executive Summary

• Dataset shape (rows × columns)
• Memory footprint
• Overall quality score
• Critical issues count

Column-by-Column Analysis

For each column, show:

• Statistics table
• Distribution histogram (ASCII for terminal)
• Top values (for categorical)
• Quality warnings

Relationships

• Correlation heatmap summary
• Potential foreign key relationships
• Column dependencies

Recommendations

• Suggested data type optimizations
• Columns to investigate
• Potential data quality rules

Output Formats

Markdown Report

Full detailed report with tables and visualizations.

JSON Summary

Machine-readable profile for programmatic use:

{
  "dataset": "data.csv",
  "rows": 50000,
  "columns": 12,
  "memory_mb": 45.2,
  "quality_score": 87.5,
  "profiles": {
    "column_name": {
      "dtype": "int64",
      "null_pct": 0.02,
      "unique_count": 45000,
      "stats": {...}
    }
  }
}

HTML Dashboard

Interactive report with charts (if ydata-profiling available).

Execution

1. Determine file type and size
2. For files > 1GB, use sampling strategy
3. Generate all statistics
4. Compile into requested output format
5. Highlight top 5 most critical findings