microcalibrate

MicroCalibrate

MicroCalibrate calibrates survey weights to match population targets, with L0 regularization for sparsity and automatic hyperparameter tuning.

For Users

What is MicroCalibrate?

When you see PolicyEngine population impacts, the underlying data has been "calibrated" using MicroCalibrate to match official population statistics.

What calibration does:

• Adjusts survey weights to match known totals (population, income, employment)
• Creates representative datasets
• Reduces dataset size while maintaining accuracy
• Ensures PolicyEngine estimates match administrative data

Example:

• Census says US has 331 million people
• Survey has 100,000 households representing the population
• MicroCalibrate adjusts weights so survey totals match census totals
• Result: More accurate PolicyEngine calculations

For Analysts

Installation

uv pip install microcalibrate

What MicroCalibrate Does

Calibration problem: You have survey data with initial weights, and you know certain population totals (benchmarks). Calibration adjusts weights so weighted survey totals match benchmarks.

Example:

from microcalibrate import Calibration
import numpy as np
import pandas as pd

# Survey data (1,000 households)
weights = np.ones(1000)  # Initial weights

# Estimates (how much each household contributes to targets)
estimate_matrix = pd.DataFrame({
    'total_income': household_incomes,      # Each household's income
    'total_employed': household_employment  # 1 if employed, 0 if not
})

# Known population targets (benchmarks)
targets = np.array([
    50_000_000,  # Total income in population
    600,         # Total employed people
])

# Calibrate
cal = Calibration(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    l0_lambda=0.01  # Sparsity penalty
)

# Optimize weights
new_weights = cal.calibrate(max_iter=1000)

# Check results
achieved = (estimate_matrix.values.T @ new_weights)
print(f"Target: {targets}")
print(f"Achieved: {achieved}")
print(f"Non-zero weights: {(new_weights > 0).sum()} / {len(weights)}")

L0 Regularization for Sparsity

Why sparsity matters:

• Reduces dataset size (fewer households to simulate)
• Faster PolicyEngine calculations
• Easier to validate and understand

L0 penalty:

# L0 encourages many weights to be exactly zero
cal = Calibration(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    l0_lambda=0.01  # Higher = more sparse
)

To see impact:

# Without L0
cal_dense = Calibration(..., l0_lambda=0.0)
weights_dense = cal_dense.calibrate()

# With L0
cal_sparse = Calibration(..., l0_lambda=0.01)
weights_sparse = cal_sparse.calibrate()

print(f"Dense: {(weights_dense > 0).sum()} households")
print(f"Sparse: {(weights_sparse > 0).sum()} households")
# Sparse might use 60% fewer households while matching same targets

Automatic Hyperparameter Tuning

Find optimal l0_lambda:

from microcalibrate import tune_hyperparameters

# Find best l0_lambda using cross-validation
best_lambda, results = tune_hyperparameters(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    lambda_min=1e-4,
    lambda_max=1e-1,
    n_trials=50
)

print(f"Best lambda: {best_lambda}")

Robustness Evaluation

Test calibration stability:

from microcalibrate import evaluate_robustness

# Holdout validation
robustness = evaluate_robustness(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix,
    l0_lambda=0.01,
    n_folds=5
)

print(f"Mean error: {robustness['mean_error']}")
print(f"Std error: {robustness['std_error']}")

Interactive Dashboard

Visualize calibration: https://microcalibrate.vercel.app/

Features:

• Upload survey data
• Set targets
• Tune hyperparameters
• View results
• Download calibrated weights

For Contributors

Repository

Location: PolicyEngine/microcalibrate

Clone:

git clone https://github.com/PolicyEngine/microcalibrate
cd microcalibrate

Current Implementation

To see structure:

tree microcalibrate/

# Key modules:
ls microcalibrate/
# - calibration.py - Main Calibration class
# - hyperparameter_tuning.py - Optuna integration
# - evaluation.py - Robustness testing
# - target_analysis.py - Target diagnostics

To see specific implementations:

# Main calibration algorithm
cat microcalibrate/calibration.py

# Hyperparameter tuning
cat microcalibrate/hyperparameter_tuning.py

# Robustness evaluation
cat microcalibrate/evaluation.py

Dependencies

Required:

• torch (PyTorch for optimization)
• l0-python (L0 regularization)
• optuna (hyperparameter tuning)
• numpy, pandas, tqdm

To see all dependencies:

cat pyproject.toml

How MicroCalibrate Uses L0

# Internal to microcalibrate
from l0 import HardConcrete

# Create gates for sample selection
gates = HardConcrete(
    n_items=len(weights),
    temperature=temperature,
    init_mean=0.999
)

# Apply gates during optimization
effective_weights = weights * gates()

# L0 penalty encourages gates → 0 or 1
# Result: Many households get weight = 0 (sparse)

To see L0 integration:

grep -n "HardConcrete\|l0" microcalibrate/calibration.py

Optimization Algorithm

Iterative reweighting:

1. Start with initial weights
2. Apply L0 gates (select samples)
3. Optimize to match targets
4. Apply penalty for sparsity
5. Iterate until convergence

Loss function:

# Target matching loss
target_loss = sum((achieved_targets - desired_targets)^2)

# L0 penalty (number of non-zero weights)
l0_penalty = l0_lambda * count_nonzero(weights)

# Total loss
total_loss = target_loss + l0_penalty

Testing

Run tests:

make test

# Or
pytest tests/ -v

To see test patterns:

cat tests/test_calibration.py
cat tests/test_hyperparameter_tuning.py

Usage in policyengine-us-data

To see how data pipeline uses microcalibrate:

cd ../policyengine-us-data

# Find usage
grep -r "microcalibrate" policyengine_us_data/
grep -r "Calibration" policyengine_us_data/

Common Patterns

Pattern 1: Basic Calibration

from microcalibrate import Calibration

cal = Calibration(
    weights=initial_weights,
    targets=benchmark_values,
    estimate_matrix=contributions,
    l0_lambda=0.01
)

calibrated_weights = cal.calibrate(max_iter=1000)

Pattern 2: With Hyperparameter Tuning

from microcalibrate import tune_hyperparameters, Calibration

# Find best lambda
best_lambda, results = tune_hyperparameters(
    weights=weights,
    targets=targets,
    estimate_matrix=estimate_matrix
)

# Use best lambda
cal = Calibration(..., l0_lambda=best_lambda)
calibrated_weights = cal.calibrate()

Pattern 3: Multi-Target Calibration

# Multiple population targets
estimate_matrix = pd.DataFrame({
    'total_population': population_counts,
    'total_income': incomes,
    'total_employed': employment_indicators,
    'total_children': child_counts
})

targets = np.array([
    331_000_000,   # US population
    15_000_000_000_000,  # Total income
    160_000_000,   # Employed people
    73_000_000     # Children
])

cal = Calibration(weights, targets, estimate_matrix, l0_lambda=0.01)

Performance Considerations

Calibration speed:

• 1,000 households, 5 targets: ~1 second
• 100,000 households, 10 targets: ~30 seconds
• Depends on: dataset size, number of targets, l0_lambda

Memory usage:

• PyTorch tensors for optimization
• Scales linearly with dataset size

To profile:

import time

start = time.time()
weights = cal.calibrate()
print(f"Calibration took {time.time() - start:.1f}s")

Troubleshooting

Common issues:

1. Calibration not converging:

# Try:
# - More iterations
# - Lower l0_lambda
# - Better initialization

cal = Calibration(..., l0_lambda=0.001)  # Lower sparsity penalty
weights = cal.calibrate(max_iter=5000)  # More iterations

2. Targets not matching:

# Check achieved vs desired
achieved = (estimate_matrix.values.T @ weights)
error = np.abs(achieved - targets) / targets
print(f"Relative errors: {error}")

# If large errors, l0_lambda may be too high

3. Too sparse (all weights zero):

# Lower l0_lambda
cal = Calibration(..., l0_lambda=0.0001)

Related Skills

• l0-skill - Understanding L0 regularization
• policyengine-us-data-skill - How calibration fits in data pipeline
• microdf-skill - Working with calibrated survey data

Resources

Repository: https://github.com/PolicyEngine/microcalibrate Dashboard: https://microcalibrate.vercel.app/ PyPI: https://pypi.org/project/microcalibrate/ Paper: Louizos et al. (2017) on L0 regularization