training

/plugin install training@aiwg      # Claude Code plugin install
aiwg use training                   # or via AIWG CLI

What aiwg-training Is

aiwg-training is a marketplace plugin for AIWG that turns any corpus — research papers, code repositories, conversation logs, documentation sites — into training-ready datasets for fine-tuning language models. It produces datasets suitable for SFT, DPO, KTO, ORPO, SimPO, and GRPO training workflows, with full provenance, license inheritance, benchmark decontamination, and byte-for-byte reproducibility.

If you have tried to build a fine-tuning dataset and ended up with ad-hoc scripts, manually curated JSONL files, mystery licenses, and hope-this-doesn't-contaminate-the-eval vibes, aiwg-training is the missing infrastructure layer. It implements every published best practice from dataset methodology research (Self-Instruct, Evol-Instruct, Orca, PersonaHub, STaR), preference-optimization research (DPO, KTO, ORPO, SimPO), governance standards (Datasheets for Datasets, Model Cards, Data Statements, ML Reproducibility Checklist), and safety research (Benchmark Contamination, Model Collapse, Llama Guard) behind a single cohesive framework.

Unlike HuggingFace datasets (storage format) or Axolotl (training orchestrator), aiwg-training is a curation pipeline. It ingests, assesses, synthesizes, filters, formats, decontaminates, versions, and documents — the work that happens before you invoke trainer.train() and the part that determines whether your fine-tune actually learns anything useful.

What Problems aiwg-training Solves

Building a fine-tuning dataset is hard in ways that don't show up in tutorials. Four failure modes dominate:

1. No Provenance, No Reproducibility

Typical dataset scripts produce JSONL files with no record of where each example came from, what license governs it, what transformations were applied, or how to rebuild the same dataset again next week. When something goes wrong — a model overfits a biased subsample, a source is later retracted, a license changes — there's no way to trace or fix it.

Without aiwg-training: 70%+ of published fine-tuning datasets fail the ML Reproducibility Checklist (Pineau et al. 2020). Lineage from raw source to trained model is almost always missing.

With aiwg-training: Every example traces back to its source via W3C PROV (REF-062). Every dataset version ships with a SHA-256 fixity manifest + deterministic seed + reproduction recipe. aiwg-training dataset reproduce byte-reproduces any prior version.

2. Benchmark Contamination

Most fine-tuning datasets accidentally include examples from the benchmarks you'll later use to evaluate the model. Your "HumanEval 67.2%" score is meaningless if 40% of HumanEval was in your training data. Published papers have been retracted over this.

Without aiwg-training: Benchmark leakage is detected post-hoc, if ever. REF-442 (Sainz et al. 2023) shows ChatGPT reproduces CoNLL-2003 verbatim — pervasive contamination across major benchmarks.

With aiwg-training: Decontamination is a first-class pipeline stage that blocks publication. Three detection modes (exact 13-gram per REF-442, fuzzy edit-distance, semantic embedding similarity). Six default targets (MMLU, GSM8K, HumanEval, HELM, MT-Bench, AlpacaEval) extensible to any benchmark. The decontamination-gate lint rule makes override explicit with triple audit trail (manifest + activity log + report appendix).

3. License Laundering