stranger-test-protocol

Stranger Test Protocol

Evaluate documentation from the perspective of someone encountering the project for the first time.

The Stranger Test

The Stranger Test asks: "Could a competent developer with zero context about this project understand what it does, why it exists, and how to get started — in under 5 minutes?"

Evaluation Framework

Level 1: First Contact (30 seconds)

Evaluate the README's first screen:

Criterion	Pass	Fail
What is it?	Clear one-sentence description	Jargon, acronyms, or missing
Who is it for?	Target audience stated or obvious	Unclear audience
Why does it exist?	Problem/value stated	No motivation
Is it alive?	Recent activity, badges, dates	No signals of maintenance

Level 2: Comprehension (2 minutes)

Evaluate understanding after reading the full README:

Criterion	Pass	Fail
Architecture	Can sketch the system on a napkin	Opaque internal terminology
Key concepts	Core abstractions defined	Assumes prior knowledge
Scope boundaries	Clear what it does and doesn't do	Ambiguous scope
Relationship to ecosystem	How it fits with other tools	Exists in a vacuum

Level 3: Activation (5 minutes)

Evaluate the getting-started experience:

Criterion	Pass	Fail
Prerequisites	Listed explicitly	Discovered through errors
Install steps	Copy-pasteable commands	Incomplete or stale
First success	Can run hello-world	Errors before first output
Next steps	Clear path forward	Dead end after install

Conducting the Test

Step 1: Adopt the Stranger Mindset

Temporarily forget all context about the project. Read as if encountering it on GitHub for the first time. Note every moment of confusion, even if you "know the answer."

Step 2: Sequential Read-Through

Read the documentation in this order, timing each section:

1. Repository name and description
2. README.md (top to bottom)
3. First linked guide (if any)
4. First code example (if any)

Step 3: Record Confusion Points

For each confusion point, record:

- **Location:** README.md, line 23
- **Confusion:** "What is an 'organ' in this context? The term is used 5 times before being defined."
- **Severity:** High (blocks understanding of everything that follows)
- **Fix:** Add a one-sentence definition before first use, or link to glossary.

Step 4: Assess Each Level

Score each criterion as PASS, PARTIAL, or FAIL. Provide evidence.

Step 5: Produce Report

## Stranger Test Report: {project-name}

**Tester:** {name/role}
**Date:** {date}
**Time to first understanding:** {seconds}
**Time to first success:** {minutes, or "did not achieve"}

### Level 1: First Contact — {PASS/PARTIAL/FAIL}
{findings}

### Level 2: Comprehension — {PASS/PARTIAL/FAIL}
{findings}

### Level 3: Activation — {PASS/PARTIAL/FAIL}
{findings}

### Confusion Points (ordered by severity)
1. {point}
2. {point}

### Recommendations (ordered by impact)
1. {recommendation}
2. {recommendation}

Common Failures

Jargon Without Definition

Bad: "This repo manages the Taxis orchestration layer for the ORGANVM eight-organ model."

Good: "This repo orchestrates automated workflows across ORGANVM — a system that organizes ~100 repositories into 8 functional groups ('organs') covering theory, art, commerce, and governance."

Missing Prerequisites

Bad:

npm install
npm start

Good:

# Prerequisites: Node.js 20+, npm 10+
# Verify: node --version && npm --version
npm install
npm start
# Open http://localhost:3000 — you should see the dashboard

Assumed Context

Bad: "See the governance rules for promotion constraints."

Good: "See governance-rules.json for promotion constraints — these define the state machine (LOCAL → CANDIDATE → GRADUATED) that every repository moves through."

Dead-End Documentation

Bad: README ends after install instructions.

Good: README includes "What's Next" section with 3-4 concrete next steps.

Applying to Different Document Types

README

Focus on Level 1 and Level 3. The README is the front door — it must answer "what" and "how to start" immediately.

API Documentation

Focus on Level 2 and Level 3. Developers need to understand concepts and make their first API call quickly.

Contributing Guide

Focus on Level 3 heavily. The reader is motivated but needs a clear path from "I want to help" to "I submitted a PR."

Architecture Docs

Focus on Level 2. Can a new team member understand the system well enough to know where to make changes?

Integration with CI

# Reminder in PR template
- [ ] Stranger test: Could a new contributor understand this PR's context from the linked docs?
- [ ] New concepts introduced in this PR are defined before first use
- [ ] Commands in documentation are copy-pasteable and tested

Anti-Patterns

• Testing your own docs — The author cannot be the stranger; fresh eyes are essential
• "It's obvious" — If it needs to be said, it's not obvious to a stranger
• Fixing symptoms — "Add a glossary" treats the symptom; "define terms on first use" treats the cause
• One-time test — Run the stranger test after every major documentation change
• Perfectionism — A PARTIAL pass with an action plan beats never testing at all