evaluation-rubrics

Evaluation Rubrics

Table of Contents

• Workflow
• Common Patterns
• Guardrails
• Quick Reference

Example

Scenario: Evaluating technical blog posts (1-5 scale)

Criterion	1 (Poor)	3 (Adequate)	5 (Excellent)
Technical Accuracy	Multiple factual errors, misleading	Mostly correct, minor inaccuracies	Fully accurate, technically rigorous
Clarity	Confusing, jargon-heavy, poor structure	Clear to experts, some structure	Accessible to target audience, well-organized
Practical Value	No actionable guidance, theoretical only	Some examples, limited applicability	Concrete examples, immediately applicable
Originality	Rehashes common knowledge, no new insight	Some fresh perspective, builds on existing	Novel approach, advances understanding

Scoring: Post A [4, 5, 3, 2] = 3.5 avg. Post B [5, 4, 5, 4] = 4.5 avg. Feedback for Post A: "Strong clarity (5) and good accuracy (4), but needs more practical examples (3) and offers less original insight (2)."

Workflow

Copy this checklist and track your progress:

Rubric Development Progress:
- [ ] Step 1: Define purpose and scope
- [ ] Step 2: Identify evaluation criteria
- [ ] Step 3: Design the scale
- [ ] Step 4: Write performance descriptors
- [ ] Step 5: Test and calibrate
- [ ] Step 6: Use and iterate

Step 1: Define purpose and scope

Clarify what you're evaluating, who evaluates, who uses results, what decisions depend on scores. See resources/template.md for scoping questions.

Step 2: Identify evaluation criteria

Brainstorm quality dimensions, prioritize most important/observable, balance coverage vs. simplicity (4-8 criteria typical). See resources/template.md for brainstorming framework.

Step 3: Design the scale

Choose number of levels (1-5, 1-4, 1-10), scale type (numeric, qualitative), anchors (what does each level mean?). See resources/methodology.md for scale selection guidance.

Step 4: Write performance descriptors

For each criterion × level, write observable description of what that performance looks like. See resources/template.md for writing guidelines.

Step 5: Test and calibrate

Have multiple reviewers score sample work, compare scores, discuss discrepancies, refine rubric. See resources/methodology.md for inter-rater reliability testing.

Step 6: Use and iterate

Apply rubric, collect feedback from evaluators and evaluatees, revise criteria/descriptors as needed. Validate using resources/evaluators/rubric_evaluation_rubrics.json. Minimum standard: Average score ≥ 3.5.

Common Patterns

Pattern 1: Analytic Rubric (Most Common)

• Structure: Multiple criteria (rows), multiple levels (columns), descriptor for each cell
• Use case: Detailed feedback needed, want to see performance across dimensions, diagnostic assessment
• Pros: Specific feedback, identifies strengths/weaknesses by criterion, high reliability
• Cons: Time-consuming to create and use, can feel reductive
• Example: Code review rubric (Correctness, Efficiency, Readability, Maintainability × 1-5 scale)

Pattern 2: Holistic Rubric

• Structure: Single overall score, descriptors integrate multiple criteria
• Use case: Quick overall judgment, summative assessment, criteria hard to separate
• Pros: Fast, intuitive, captures gestalt quality
• Cons: Less actionable feedback, lower reliability, can't diagnose specific weaknesses
• Example: Essay holistic scoring (1=poor essay, 3=adequate essay, 5=excellent essay with detailed descriptors)

Pattern 3: Single-Point Rubric

• Structure: Criteria listed with only "meets standard" descriptor, space to note above/below
• Use case: Growth mindset feedback, encourage self-assessment, less punitive feel
• Pros: Emphasizes improvement not deficit, simpler to create, encourages dialogue
• Cons: Less precision, requires written feedback to supplement
• Example: Design critique (list criteria like "Visual hierarchy", "Accessibility", note "+Clear focal point, -Poor contrast")

Pattern 4: Checklist (Binary)

• Structure: List of yes/no items, must-haves for acceptance
• Use case: Compliance checks, minimum quality gates, pass/fail decisions
• Pros: Very clear, objective, easy to use
• Cons: No gradations, misses quality beyond basics, can feel rigid
• Example: Pull request checklist (Tests pass? Code linted? Documentation updated? Security review?)

Pattern 5: Standards-Based Rubric

• Structure: Criteria tied to learning objectives/competencies, levels = degree of mastery
• Use case: Educational assessment, skill certification, training evaluation, criterion-referenced
• Pros: Aligned to standards, shows progress toward mastery, diagnostic
• Cons: Requires clear standards, can be complex to design
• Example: Data science skills (Proficiency in: Data cleaning, Modeling, Visualization, Communication × Novice/Competent/Expert)

Guardrails

1. Criteria should be observable and measurable: Not "good attitude" (subjective), but "arrives on time, volunteers for tasks, helps teammates" (observable). Test: Can two independent reviewers score this criterion consistently?

2. Descriptors should distinguish levels clearly: Each level needs concrete differences from adjacent levels. Avoid "5=very good, 4=good, 3=okay". Better: "5=zero bugs, meets all requirements, 4=1-2 minor bugs, meets 90% requirements."

3. Use appropriate scale granularity: 1-3 is too coarse, 1-10 is too fine. Sweet spot: 1-4 (forced choice, no middle) or 1-5 (allows neutral middle). Match granularity to actual observable differences.

4. Balance comprehensiveness with simplicity: Aim for 4-8 criteria covering essential quality dimensions. If >10 criteria, consider grouping or prioritizing.

5. Calibrate for inter-rater reliability: Have multiple reviewers score same work, measure agreement (Kappa, ICC). If <70% agreement, refine descriptors.

6. Provide examples at each level: Include concrete examples of work at each level (anchor papers, reference designs, code samples) to calibrate reviewers.

7. Share rubric before evaluation: If evaluatees see the rubric only after being scored, it is grading not guidance. Share upfront so people know expectations and can self-assess.

8. Weight criteria appropriately: If "Security" matters more than "Code style", weight it (Security x3, Style x1). Or use thresholds (score >=4 on Security to pass, regardless of other scores).

Common pitfalls:

• ❌ Subjective language: "Shows effort", "creative", "professional" - not observable without concrete descriptors
• ❌ Overlapping criteria: "Clarity" and "Organization" often conflated - define boundaries clearly
• ❌ Hidden expectations: Rubric doesn't mention X, but evaluators penalize for missing X - document all criteria
• ❌ Central tendency bias: Reviewers avoid extremes (always score 3/5) - use even-number scales (1-4) to force choice
• ❌ Halo effect: High score on one criterion biases other scores up - score each criterion independently before looking at others
• ❌ Rubric drift: Descriptors erode over time, reviewers interpret differently - periodic re-calibration required

Quick Reference

Key resources:

• resources/template.md: Purpose definition, criteria brainstorming, scale selection, descriptor templates, rubric formats
• resources/methodology.md: Scale design principles, descriptor writing techniques, inter-rater reliability testing, bias mitigation
• resources/evaluators/rubric_evaluation_rubrics.json: Quality criteria for rubric design (criteria clarity, scale appropriateness, descriptor specificity)

Scale Selection Guide:

Scale	Use When	Pros	Cons
1-3	Need quick categorization, clear tiers	Fast, forces clear decision	Too coarse, less feedback
1-4	Want forced choice (no middle)	Avoids central tendency, clear differentiation	No neutral option, feels binary
1-5	General purpose, most common	Allows neutral, familiar, good granularity	Central tendency bias (everyone gets 3)
1-10	Need fine gradations, large sample	Maximum differentiation, statistical analysis	False precision, hard to distinguish adjacent levels
Qualitative (Novice/Proficient/Expert)	Educational, skill development	Intuitive, growth-oriented	Less quantitative, harder to aggregate
Binary (Yes/No, Pass/Fail)	Compliance, gatekeeping	Objective, simple	No gradations, misses quality differences

Criteria Types:

• Product criteria: Evaluate the artifact itself (correctness, clarity, completeness, aesthetics, performance)
• Process criteria: How work was done (methodology followed, collaboration, iteration, time management)
• Impact criteria: Outcomes/effects (user satisfaction, business value, learning achieved)
• Meta criteria: Quality of quality (documentation, testability, maintainability, scalability)

Inter-Rater Reliability Benchmarks:

• <50% agreement: Rubric unreliable, needs major revision
• 50-70% agreement: Marginal, refine descriptors and calibrate reviewers
• 70-85% agreement: Good, acceptable for most uses
• >85% agreement: Excellent, highly reliable scoring

Typical Rubric Development Time:

• Simple rubric (3-5 criteria, 1-4 scale, known domain): 2-4 hours
• Standard rubric (5-7 criteria, 1-5 scale, some complexity): 6-10 hours + calibration session
• Complex rubric (8+ criteria, multiple scales, novel domain): 15-25 hours + multiple calibration rounds

When to escalate beyond rubrics:

• High-stakes decisions (hiring, admissions, awards) → Add structured interviews, portfolios, multi-method assessment
• Subjective/creative work (art, poetry, design) → Supplement rubric with critique, discourse, expert judgment
• Complex holistic judgment (leadership, cultural fit) → Rubrics help but don't capture everything, use thoughtfully → Rubrics are tools not replacements for human judgment. Use to structure thinking, not mechanize decisions.

Inputs required:

• Artifact type (what are we evaluating? essays, code, designs, proposals?)
• Criteria (quality dimensions to assess, 4-8 most common)
• Scale (1-5 default, or specify 1-4, 1-10, qualitative labels)

Outputs produced:

• evaluation-rubrics.md: Purpose, criteria definitions, scale with descriptors, usage instructions, weighting/thresholds, calibration notes