Golden Dataset

Comprehensive patterns for building, managing, and validating golden datasets for AI/ML evaluation. Each category has individual rule files in rules/ loaded on-demand.

Quick Reference

Category	Rules	Impact	When to Use
Curation	3	HIGH	Content collection, annotation pipelines, diversity analysis
Management	3	HIGH	Versioning, backup/restore, CI/CD automation
Validation	3	CRITICAL	Quality scoring, drift detection, regression testing
Add Workflow	1	HIGH	9-phase curation, quality scoring, bias detection, silver-to-gold

Total: 10 rules across 4 categories

Curation

Content collection, multi-agent annotation, and diversity analysis for golden datasets.

Rule	File	Key Pattern
Collection	rules/curation-collection.md	Content type classification, quality thresholds, duplicate prevention
Annotation	rules/curation-annotation.md	Multi-agent pipeline, consensus aggregation, Langfuse tracing
Diversity	rules/curation-diversity.md	Difficulty stratification, domain coverage, balance guidelines

Management

Versioning, storage, and CI/CD automation for golden datasets.

Rule	File	Key Pattern
Versioning	rules/management-versioning.md	JSON backup format, embedding regeneration, disaster recovery
Storage	rules/management-storage.md	Backup strategies, URL contract, data integrity checks
CI Integration	rules/management-ci.md	GitHub Actions automation, pre-deployment validation, weekly backups

Validation

Quality scoring, drift detection, and regression testing for golden datasets.

Rule	File	Key Pattern
Quality	rules/validation-quality.md	Schema validation, content quality, referential integrity
Drift	rules/validation-drift.md	Duplicate detection, semantic similarity, coverage gap analysis
Regression	rules/validation-regression.md	Difficulty distribution, pre-commit hooks, full dataset validation

Add Workflow

Structured workflow for adding new documents to the golden dataset.

Rule	File	Key Pattern
Add Document	rules/curation-add-workflow.md	9-phase curation, parallel quality analysis, bias detection

Quick Start Example

from app.shared.services.embeddings import embed_text

async def validate_before_add(document: dict, source_url_map: dict) -> dict:
    """Pre-addition validation for golden dataset entries."""
    errors = []

    # 1. URL contract check
    if "placeholder" in document.get("source_url", ""):
        errors.append("URL must be canonical, not a placeholder")

    # 2. Content quality
    if len(document.get("title", "")) < 10:
        errors.append("Title too short (min 10 chars)")

    # 3. Tag requirements
    if len(document.get("tags", [])) < 2:
        errors.append("At least 2 domain tags required")

    return {"valid": len(errors) == 0, "errors": errors}

Key Decisions

Decision	Recommendation
Backup format	JSON (version controlled, portable)
Embedding storage	Exclude from backup (regenerate on restore)
Quality threshold	>= 0.70 quality score for inclusion
Confidence threshold	>= 0.65 for auto-include
Duplicate threshold	>= 0.90 similarity blocks, >= 0.85 warns
Min tags per entry	2 domain tags
Min test queries	3 per document
Difficulty balance	Trivial 3, Easy 3, Medium 5, Hard 3 minimum
CI frequency	Weekly automated backup (Sunday 2am UTC)

Common Mistakes

1. Using placeholder URLs instead of canonical source URLs
2. Skipping embedding regeneration after restore
3. Not validating referential integrity between documents and queries
4. Over-indexing on articles (neglecting tutorials, research papers)
5. Missing difficulty distribution balance in test queries
6. Not running verification after backup/restore operations
7. Testing restore procedures in production instead of staging
8. Committing SQL dumps instead of JSON (not version-control friendly)

Evaluations

See test-cases.json for 9 test cases across all categories.

Related Skills

• ork:rag-retrieval - Retrieval evaluation using golden dataset
• langfuse-observability - Tracing patterns for curation workflows
• ork:testing-unit - Unit testing patterns and strategies
• ai-native-development - Embedding generation for restore

Capability Details

curation

Keywords: golden dataset, curation, content collection, annotation, quality criteria

Solves:

• Classify document content types for golden dataset
• Run multi-agent quality analysis pipelines
• Generate test queries for new documents

management

Keywords: golden dataset, backup, restore, versioning, disaster recovery

Solves:

• Backup and restore golden datasets with JSON
• Regenerate embeddings after restore
• Automate backups with CI/CD

validation

Keywords: golden dataset, validation, schema, duplicate detection, quality metrics

Solves:

• Validate entries against document schema
• Detect duplicate or near-duplicate entries
• Analyze dataset coverage and distribution gaps