Code Sample Validation Skill

Extract, validate, and test code samples in documentation.

Capabilities

• Extract code blocks from Markdown/MDX
• Syntax validation for multiple languages
• Execute and verify code sample output
• Snippet compilation testing
• Version compatibility checking
• Import/dependency verification
• Code formatting validation (prettier, black)
• Generate runnable code from documentation

Usage

Invoke this skill when you need to:

• Validate code samples in documentation
• Test code examples execute correctly
• Check for outdated API usage
• Verify import statements
• Generate test files from documentation

Inputs

Parameter	Type	Required	Description
inputPath	string	Yes	Path to documentation file or directory
action	string	Yes	extract, validate, execute, format
languages	array	No	Filter by language (js, python, etc.)
outputDir	string	No	Directory for extracted runnable code
timeout	number	No	Execution timeout in seconds
config	object	No	Language-specific configuration

Input Example

{
  "inputPath": "./docs",
  "action": "validate",
  "languages": ["javascript", "python"],
  "timeout": 30
}

Output Structure

Validation Report

{
  "summary": {
    "total": 45,
    "passed": 42,
    "failed": 2,
    "skipped": 1
  },
  "files": [
    {
      "file": "docs/quickstart.md",
      "samples": [
        {
          "language": "javascript",
          "line": 15,
          "status": "passed",
          "syntaxValid": true,
          "executionResult": {
            "success": true,
            "output": "Hello, World!",
            "duration": 125
          }
        },
        {
          "language": "python",
          "line": 45,
          "status": "failed",
          "syntaxValid": true,
          "executionResult": {
            "success": false,
            "error": "ModuleNotFoundError: No module named 'requests'",
            "suggestion": "Add 'requests' to test dependencies"
          }
        }
      ]
    }
  ],
  "issues": [
    {
      "file": "docs/api/users.md",
      "line": 78,
      "language": "typescript",
      "issue": "Type error: Property 'user' does not exist on type 'Response'",
      "code": "const name = response.user.name;",
      "suggestion": "Update to use 'response.data.user.name'"
    }
  ]
}

Code Extraction

Markdown Code Block Patterns

```javascript
// This block will be extracted and validated
const client = new Client({ apiKey: 'test' });
const result = await client.query('Hello');
console.log(result);
```

```python
# This block will be extracted and validated
from mypackage import Client

client = Client(api_key='test')
result = client.query('Hello')
print(result)
```

```bash skip-validation
# This block will be skipped due to directive
echo "Not validated"
```

```javascript title="example.js" runnable
// This block is marked as runnable
export function greet(name) {
  return `Hello, ${name}!`;
}
```

Extraction Configuration

{
  "extract": {
    "includeLanguages": ["javascript", "typescript", "python", "go"],
    "excludeLanguages": ["bash", "shell", "text"],
    "directives": {
      "skip": ["skip-validation", "no-test"],
      "runnable": ["runnable", "test"],
      "expectError": ["expect-error"]
    },
    "metaPatterns": {
      "title": "title=\"([^\"]+)\"",
      "filename": "filename=\"([^\"]+)\""
    }
  }
}

Language-Specific Validation

JavaScript/TypeScript

// validator-config.js
module.exports = {
  javascript: {
    parser: 'babel',
    parserOptions: {
      ecmaVersion: 2024,
      sourceType: 'module'
    },
    execute: {
      runtime: 'node',
      timeout: 10000,
      setup: `
        global.fetch = require('node-fetch');
        process.env.API_KEY = 'test-key';
      `
    },
    format: {
      tool: 'prettier',
      config: {
        semi: true,
        singleQuote: true,
        trailingComma: 'es5'
      }
    }
  },
  typescript: {
    compiler: 'tsc',
    compilerOptions: {
      target: 'ES2022',
      module: 'ESNext',
      strict: true
    },
    execute: {
      runtime: 'ts-node',
      timeout: 15000
    }
  }
};

Python

# validator_config.py
PYTHON_CONFIG = {
    'version': '3.11',
    'execute': {
        'timeout': 30,
        'setup': '''
import os
os.environ['API_KEY'] = 'test-key'
''',
        'virtualenv': '.venv'
    },
    'format': {
        'tool': 'black',
        'config': {
            'line_length': 88,
            'target_version': ['py311']
        }
    },
    'lint': {
        'tool': 'ruff',
        'rules': ['E', 'F', 'W']
    }
}

Go

// validator_config.go
package main

var GoConfig = Config{
    Version: "1.21",
    Execute: ExecuteConfig{
        Timeout: 30,
        Build:   true,
        Run:     true,
    },
    Format: FormatConfig{
        Tool: "gofmt",
    },
    Lint: LintConfig{
        Tool:  "golangci-lint",
        Rules: []string{"govet", "errcheck", "staticcheck"},
    },
}

Test Generation

Generate Test Files

// Generated from docs/quickstart.md

const { Client } = require('@example/sdk');

describe('Documentation Code Samples', () => {
  describe('quickstart.md', () => {
    test('Line 15: Basic client usage', async () => {
      const client = new Client({ apiKey: 'test' });
      const result = await client.query('Hello');
      expect(result).toBeDefined();
    });

    test('Line 45: Error handling', async () => {
      const client = new Client({ apiKey: 'invalid' });
      await expect(client.query('Hello')).rejects.toThrow('Invalid API key');
    });
  });
});

Test Configuration

{
  "testGeneration": {
    "framework": "jest",
    "outputDir": "tests/docs",
    "filePattern": "{docfile}.test.js",
    "imports": [
      "const { Client } = require('@example/sdk');",
      "const { mockServer } = require('./mocks');"
    ],
    "beforeAll": "await mockServer.start();",
    "afterAll": "await mockServer.stop();",
    "assertions": {
      "outputMatch": true,
      "noThrow": true,
      "typeCheck": true
    }
  }
}

CI/CD Integration

GitHub Actions

name: Validate Documentation

on:
  push:
    paths:
      - 'docs/**/*.md'
  pull_request:
    paths:
      - 'docs/**/*.md'

jobs:
  validate-code-samples:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          npm ci
          pip install -r requirements-docs.txt

      - name: Validate code samples
        run: |
          node scripts/validate-docs.js \
            --input docs/ \
            --languages javascript,python \
            --report validation-report.json

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: validation-report
          path: validation-report.json

      - name: Check for failures
        run: |
          if jq '.summary.failed > 0' validation-report.json | grep -q true; then
            echo "Code sample validation failed"
            jq '.issues' validation-report.json
            exit 1
          fi

Formatting Validation

Prettier Check

const prettier = require('prettier');

async function validateFormatting(code, language) {
  const options = {
    parser: getParser(language),
    semi: true,
    singleQuote: true,
    trailingComma: 'es5',
  };

  try {
    const formatted = await prettier.format(code, options);
    const isFormatted = code === formatted;

    return {
      valid: isFormatted,
      formatted: formatted,
      diff: isFormatted ? null : generateDiff(code, formatted),
    };
  } catch (error) {
    return {
      valid: false,
      error: error.message,
    };
  }
}

Black Check (Python)

import black
import difflib

def validate_formatting(code: str) -> dict:
    try:
        mode = black.Mode(
            target_versions={black.TargetVersion.PY311},
            line_length=88,
        )
        formatted = black.format_str(code, mode=mode)
        is_formatted = code == formatted

        return {
            'valid': is_formatted,
            'formatted': formatted,
            'diff': None if is_formatted else list(
                difflib.unified_diff(
                    code.splitlines(),
                    formatted.splitlines(),
                    lineterm=''
                )
            )
        }
    except black.InvalidInput as e:
        return {
            'valid': False,
            'error': str(e)
        }

Workflow

1. Extract samples - Parse Markdown for code blocks
2. Filter - Apply language and directive filters
3. Validate syntax - Check for syntax errors
4. Check imports - Verify dependencies exist
5. Execute - Run samples in sandbox
6. Verify output - Check expected results
7. Check formatting - Validate code style
8. Report - Generate validation report

Dependencies

{
  "devDependencies": {
    "@babel/parser": "^7.23.0",
    "prettier": "^3.0.0",
    "typescript": "^5.3.0",
    "ts-node": "^10.9.0",
    "jest": "^29.7.0",
    "gray-matter": "^4.0.0"
  }
}

CLI Commands

# Validate all samples
node scripts/validate-docs.js --input docs/

# Extract and save runnable code
node scripts/validate-docs.js --input docs/ --action extract --output tests/samples/

# Generate test files
node scripts/validate-docs.js --input docs/ --action generate-tests --output tests/docs/

# Check formatting only
node scripts/validate-docs.js --input docs/ --action format-check

Best Practices Applied

• Mark non-runnable samples with directives
• Include expected output for verification
• Use consistent formatting across samples
• Pin dependency versions in samples
• Test samples in CI/CD pipeline
• Generate runnable tests from docs

References

• Prettier: https://prettier.io/
• Black: https://black.readthedocs.io/
• MDX: https://mdxjs.com/
• Jest: https://jestjs.io/

Target Processes

• docs-testing.js
• api-reference-docs.js
• sdk-doc-generation.js
• interactive-tutorials.js