stage1-data-pipeline

Stage 1 · Skeleton · Data pipeline (leakage-safe)

The single most under-respected rule in deep learning: fit every transform on training data only, ever. Violating this rule is the most common silent way to produce results that don't replicate.

Stage question

"Could information from validation or test data have leaked into the training pipeline?"

If you cannot answer "no, by construction", you are not yet through Stage 1.

The four-step pipeline contract

Every dataset adapter in curryTrain follows this order, always:

raw → split → fit_on_train_only → apply_to_all → load_loader

1. Raw: load files, decode, but do nothing else. No filtering, no normalization, no tokenization.
2. Split: separate into train / val / test by some deterministic rule. For temporal data, time-aware: train < val < test by timestamp, never by random shuffle.
3. Fit on train only: any object that has parameters (normalizer, tokenizer, label encoder, vocabulary, feature selector) is .fit(...) on the train split alone, then frozen.
4. Apply to all: the fitted object is applied to all splits.
5. Loader: per-split DataLoader with appropriate batch size, shuffle on train only, deterministic seed.

If any of these steps is out of order, the pipeline is leaking.

Time-aware split rules (when applicable)

For data with timestamps:

• Train: t < t_split_train_val
• Val: t_split_train_val ≤ t < t_split_val_test
• Test: t ≥ t_split_val_test

Never do random_split on temporal data. Never use any future information (e.g. moving averages computed over all time) in a feature.

Deterministic shuffle and seed

The dataloader for the train split shuffles. Use a Generator seeded with the run's seed so the shuffle is reproducible:

g = torch.Generator()
g.manual_seed(seed)
DataLoader(train_set, shuffle=True, generator=g, ...)

Validation and test loaders never shuffle.

Preflight check against leakage

This is assert_no_leak_in_data_pipeline from stage1-preflight-asserts. The probe:

1. Take 10 samples from the train pipeline. Record their transformed values.
2. Take 10 samples from the val pipeline. Record their transformed values.
3. Refit the transform on val and apply to the same 10 train samples.
4. If the train values changed, the transform was using val statistics → fail.

This catches the common bug of StandardScaler().fit(full_dataset) instead of .fit(train).

Curriculum / data ordering (later stages, not Stage 1)

Curriculum learning, data deduplication, quality filtering — these belong to Stage 4 Scale-up at the earliest. In Stage 1, just produce a clean, leakage-safe pipeline. Don't optimize the data ordering until you've proven the model works at all.

Procedure when assisting a user

1. Ask the user for their split rule. If they cannot articulate it, that's the first problem to solve.

2. Inspect their pipeline code. Look for .fit( calls. Every .fit call must be on a train-only subset.

3. If they have temporal data, check that the split is time-aware. Random splits on temporal data are a frequent bug.

4. Wire up assert_no_leak_in_data_pipeline in their preflight. Run it once. Failures here block Stage 2.

5. Confirm the train DataLoader is the only one with shuffle=True and the only one using a seeded Generator.

Boundaries

• This skill is about correctness, not throughput. Mosaic Streaming, WebDataset, sharded readers — all of those belong to Stage 4 (Scale-up) when throughput matters.
• This skill does not cover augmentation choices. Augmentation tuning is a Stage 3 (Pre-validate) concern.
• This skill does not cover tokenization — see whatever the model package's tokenizer dependency is.

Common failure modes to flag

• scaler.fit(X_all) followed by per-split application → leakage. Re-do as scaler.fit(X_train).
• train_test_split(X, y, random_state=...) on temporal data → time leak.
• Vocabulary built from train + val → val signal leaks into the embedding matrix.
• Augmentations applied identically to train and val → val performance becomes biased.

Related

• skills/stage1-preflight-asserts (assert 6 specifically).
• skills/stage2-overfit-single-batch — once leakage is ruled out, sanity check the model.
• Andrew Ng's "Train/Dev/Test set distribution" chapter in Machine Learning Yearning.