Harness DX

Harness DX

Audit developer experience artifacts -- README quality, API documentation coverage, getting-started guides, and example code validation. Produces a structured DX scorecard with specific improvements and scaffolds missing documentation.

When to Use

• When preparing a library, SDK, or open-source project for release and developer adoption matters
• When reviewing a PR that changes public API surface and documentation should match
• When onboarding friction is high and you need to identify where developers get stuck
• NOT for internal architecture documentation (use harness-docs-pipeline)
• NOT for user-facing product copy (use harness-ux-copy)
• NOT for API design decisions like REST vs GraphQL (use harness-api-design)

Process

Phase 1: AUDIT -- Evaluate Documentation Quality

1. Resolve project root. Use provided path or cwd.

2. Locate documentation artifacts. Search for:

   • README files: README.md, README.rst, readme.md
   • Getting started: GETTING_STARTED.md, QUICKSTART.md, docs/getting-started.md
   • API docs: docs/api/, API.md, generated docs in docs/, site/
   • Examples: examples/, demos/, samples/, code blocks in README
   • Changelog: CHANGELOG.md, CHANGES.md, HISTORY.md
   • Contributing: CONTRIBUTING.md, .github/CONTRIBUTING.md

3. Score README completeness. Check for the presence and quality of each section:

   • Title and description (what is this project?) -- 0-2 points
   • Installation/setup (how do I get it?) -- 0-3 points
   • Quick example (show me it working in under 30 seconds) -- 0-3 points
   • API reference or link (where is the full documentation?) -- 0-2 points
   • Contributing guide or link -- 0-1 point
   • License -- 0-1 point
   • Total: score out of 12, grade A (10+), B (7-9), C (4-6), D (0-3)

4. Evaluate installation instructions. Check:

   • Are all package managers covered? (npm, yarn, pnpm for JS; pip, poetry for Python; cargo for Rust)
   • Are prerequisites listed? (Node version, OS requirements, system dependencies)
   • Is there a one-liner to get started? (copy-paste friendly)
   • Do the instructions work on all documented platforms?

5. Assess API documentation coverage. For every exported function, class, or endpoint:

   • Is it documented?
   • Does it have parameter descriptions?
   • Does it have a usage example?
   • Does it have return type documentation?
   • Calculate coverage percentage: documented / total * 100

6. Check for time-to-hello-world. Estimate the number of steps from git clone to seeing the project work. Fewer than 5 steps is good. More than 10 is a problem.

---

Phase 2: EXTRACT -- Identify and Validate Examples

1. Extract code examples from documentation. Parse all markdown files for fenced code blocks with language annotations. Track:

   • File location and line number
   • Language (js, ts, python, bash, etc.)
   • Whether it is a complete runnable example or a fragment

2. Extract standalone examples. Scan examples/, demos/, samples/ for:

   • Example projects with their own package.json/requirements.txt
   • Single-file examples
   • Example README files explaining what each example demonstrates

3. Validate example syntax. For each extracted code example:

   • Check for syntax errors (missing imports, unclosed brackets, invalid syntax)
   • Check for references to APIs that no longer exist (stale examples)
   • Check that import paths match the actual package name and exports

4. Run executable examples. When --validate-examples is set:

   • For JavaScript/TypeScript: attempt node or tsx execution
   • For Python: attempt python execution
   • For shell commands: validate they reference real scripts and flags
   • Record pass/fail for each example with error output

5. Check example freshness. Compare examples against the current API surface:

   • Are there deprecated APIs used in examples?
   • Are there new APIs with no examples?
   • When was each example file last modified relative to the source it demonstrates?

6. Build coverage map. Map examples to the APIs they demonstrate. Identify APIs with zero examples (documentation gaps).

---

Phase 3: SCAFFOLD -- Generate Missing Documentation

1. Generate README sections. For any missing README section identified in Phase 1:

   • Draft installation instructions by reading package.json, setup.py, Cargo.toml, or equivalent
   • Draft a quick-start example using the project's main export
   • Draft a features list from the project's exports and test descriptions

2. Generate API documentation stubs. For undocumented exports:

   • Extract function signatures, parameter types, and return types from source
   • Generate JSDoc/docstring stubs with parameter descriptions inferred from type names
   • Include a usage example skeleton derived from test files when available

3. Generate example files. For APIs with no examples:

   • Create a minimal working example in examples/
   • Include comments explaining each step
   • Ensure the example is self-contained (includes imports, setup, and cleanup)

4. Generate getting-started guide. If no quickstart exists:

   • Write a step-by-step guide from installation through first meaningful use
   • Include expected output at each step
   • Target under 5 minutes to complete

5. Propose documentation structure. If documentation is scattered or missing:

   • Recommend a docs/ directory structure
   • Map content to sections (guides, reference, examples, tutorials)
   • Suggest a documentation site generator if the project is large enough (Docusaurus, MkDocs, mdBook)

---

Phase 4: VALIDATE -- Verify Documentation Accuracy

1. Check link integrity. Verify all links in documentation:

   • Internal links: do referenced files and anchors exist?
   • External links: are they well-formed? (do not make HTTP requests)
   • Badge URLs: are shields.io and similar badge URLs using the correct repo/package name?

2. Check version consistency. Verify documentation matches the current version:

   • Does the installation section reference the correct package version?
   • Do API examples use the current function signatures?
   • Is the changelog up to date with the latest release?

3. Check cross-references. Verify README links to detailed docs, and detailed docs link back to the README and to each other where appropriate.

4. Output DX scorecard. Present the complete audit results:

   DX Scorecard: [GRADE]
   README: [score]/12 ([grade])
   API Coverage: [N]% ([documented]/[total] exports)
   Examples: [working]/[total] passing
   Time to Hello World: ~[N] steps
   Links: [valid]/[total] verified
   
   GAPS:
   - Missing: getting-started guide
   - Missing: 12 undocumented exports
   - Broken: examples/advanced.ts references removed API
   
   GENERATED:
   - docs/getting-started.md (draft)
   - 4 API documentation stubs added
   - examples/basic-usage.ts created
   

5. Verify scaffolded content compiles. If documentation was generated, verify:

   • Generated code examples have valid syntax
   • Generated markdown renders correctly (no broken formatting)
   • Generated files are placed in the correct directories

---

Harness Integration

• harness skill run harness-dx -- Primary command for running the DX audit.
• harness validate -- Run after scaffolding documentation to verify project health.
• Glob -- Used to locate README files, documentation directories, example folders, and API docs.
• Grep -- Used to extract exported symbols, find documentation comments, and locate code examples in markdown.
• Read -- Used to read documentation files, package manifests, and source files for API extraction.
• Write -- Used to scaffold missing documentation, generate example files, and create getting-started guides.
• Bash -- Used to run example validation, check link targets, and execute code snippets.
• emit_interaction -- Used to present the DX scorecard and request confirmation before generating scaffolded files.

Success Criteria

• README is scored against all 6 completeness criteria with specific gap identification
• API documentation coverage percentage is calculated against actual exported surface
• All code examples in documentation are syntax-checked
• Executable examples pass when --validate-examples is set
• Missing documentation is scaffolded with accurate, runnable content
• DX scorecard provides an at-a-glance quality grade
• Time-to-hello-world is estimated and actionable if too high

Examples

Example: Node.js SDK with Sparse Documentation

Phase 1: AUDIT
  README score: 5/12 (C)
    Present: title, description, license
    Missing: installation, quick example, API reference link, contributing
  API coverage: 23% (7/30 exports documented)
  Time to hello world: ~14 steps (too many, target: <5)

Phase 2: EXTRACT
  Code examples found: 3 (all in README)
  examples/ directory: empty
  Validation: 2/3 examples pass syntax check
  Broken: README line 45 references `sdk.connect()` -- renamed to `sdk.init()` in v2.0

Phase 3: SCAFFOLD
  Generated: docs/getting-started.md (5-step quickstart)
  Generated: examples/basic-usage.ts (demonstrates init, query, cleanup)
  Generated: 23 JSDoc stubs from TypeScript signatures
  README patches: added installation section, updated broken example

Phase 4: VALIDATE
  Links: 8/10 valid (2 broken anchors in README)
  Generated examples: syntax valid
  DX Scorecard: C -> B (projected after applying changes)

Example: Python Library with Comprehensive Docs (Sphinx)

Phase 1: AUDIT
  README score: 11/12 (A)
    Missing only: contributing guide link
  API coverage: 89% (142/160 functions documented)
  Sphinx docs at docs/_build/html: present, 45 pages
  Time to hello world: ~4 steps (good)

Phase 2: EXTRACT
  Code examples: 28 in docs, 12 in examples/
  Validation: 37/40 pass (3 use deprecated pandas.append)
  Stale examples: 3 files last modified 8 months ago, source changed since

Phase 3: SCAFFOLD
  Generated: 18 docstring stubs for undocumented functions
  Updated: 3 stale examples to use pandas.concat
  Added: CONTRIBUTING.md link to README

Phase 4: VALIDATE
  Links: 52/52 valid
  DX Scorecard: A (maintained, minor freshness issues resolved)

Example: Rust CLI Tool Missing Getting Started

Phase 1: AUDIT
  README score: 7/12 (B)
    Present: title, description, installation (cargo install), license, API link
    Missing: quick example showing actual CLI usage, contributing
  API coverage: N/A (CLI tool, not library)
  CLI help text: present via clap derive
  Time to hello world: ~6 steps

Phase 2: EXTRACT
  Code examples: 2 in README (both installation commands)
  examples/ directory: 1 example config file, no runnable examples
  Missing: actual usage examples showing command output

Phase 3: SCAFFOLD
  Generated: docs/getting-started.md with:
    1. cargo install myctl
    2. myctl init
    3. myctl run --config example.toml
    (with expected output at each step)
  Generated: examples/basic-config.toml with annotated comments
  Generated: README quick-example section with terminal output

Phase 4: VALIDATE
  CLI help flags match documented flags: YES
  Config example matches current schema: YES
  DX Scorecard: B -> A (projected after applying changes)

Rationalizations to Reject

Rationalization	Reality
"The README has an installation section but it only covers npm — yarn and pnpm users can figure it out. I'll mark installation as complete."	Installation instructions must cover all package managers the project supports. If yarn.lock or pnpm-lock.yaml exists alongside package-lock.json, all three installers must be documented. Partial coverage is scored as partial, not complete.
"This code example in the README uses the old sdk.connect() API — but it still parses syntactically, so it passes the syntax check."	Stale API references are broken examples regardless of syntax validity. A syntactically valid example that calls a renamed or removed function fails the freshness check and must be flagged as broken in the scorecard.
"The API function's behavior is complex, but I can infer what it does from the name parseAndValidate — I'll write the docstring stub based on that."	Documentation must be derived from actual source code: type signatures, test files, and existing docs. Inferring behavior from function names produces fabricated documentation. Flag functions that cannot be documented from source as requiring developer-written docs.
"The getting-started guide already exists in the wiki — it's not in the repo, but I'll mark the quickstart as present."	Documentation must be locatable from the repository root. A wiki link from the README satisfies the API reference link criterion only if the link is explicit. A guide that requires knowing where the wiki is does not meet the discoverability requirement.
"There are 18 undocumented exports — I'll generate all 18 JSDoc stubs and commit them without showing the user first."	Scaffolded documentation must be presented for review before being written. Generated stubs may contain inaccurate parameter descriptions or wrong return type assumptions. Use emit_interaction to present scaffolded content and wait for approval.

Gates

• No scaffolding without human confirmation. Generated documentation is always presented as a draft for review. Do not commit generated files automatically. Use emit_interaction to present scaffolded content and wait for approval.
• No overwriting existing documentation. If a README section already exists, do not replace it. Only fill gaps. Existing content may have been carefully written and should not be clobbered.
• No fabricating API behavior. Generated documentation and examples must be derived from actual source code (type signatures, test files, existing docs). Do not guess what an undocumented function does based on its name alone.
• No marking stale examples as passing. If an example references a renamed or removed API, it is broken regardless of whether it happens to still parse syntactically.

Escalation

• When API documentation requires domain expertise: If function behavior cannot be inferred from types and tests alone, flag it: "These 5 functions need developer-written documentation -- their behavior is domain-specific and cannot be reliably inferred."
• When examples require external services: If running an example requires a database, API key, or external service, flag the dependency rather than failing: "This example requires a running PostgreSQL instance. Consider adding a Docker Compose file for example dependencies."
• When documentation tooling is broken: If Sphinx, TypeDoc, or other doc generators fail to build, report the error but do not attempt to fix the toolchain. That is outside this skill's scope.
• When README and API docs contradict each other: Flag the contradiction with both sources quoted. Do not choose which one is correct -- the developer must resolve the conflict: "README says init() accepts a string, but the TypeDoc shows it accepts InitConfig. Which is current?"