answer-processing

Answer Processing

When to load

• User uploads an answer PDF and asks to grade it
• /grade is invoked
• User says "I finished the quiz, here's my work"

Core pipeline

answers/<quiz-name>.pdf      ← user uploads hand-written scan
      ↓ (pdf skill, OCR)
answers/converted/<quiz-name>.md
      ↓ (this skill)
grade report → stdout (compact) + errors/log.md (append)

Step-by-step procedure

Step 1: Locate the answer file

If /grade was called with an argument, use it as a hint. Otherwise find the most recently modified file in answers/ (not answers/converted/).

Step 2: Convert PDF to MD (if PDF)

Use the vision-ocr skill — delegates to a local VLM (Qwen3-VL 8B via ollama) for clean prose + LaTeX transcription (the script reads INTERFACE_LANG from .course-meta so the VLM keeps the handwriting in its original language), with pytesseract as automatic fallback.

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/vision_ocr.py" answers/<name>.pdf answers/converted/<name>.md

The script handles model warmup, page-by-page inference, and tier fallback. See .claude/skills/vision-ocr/SKILL.md. The output header tells the grader which tier produced the text:

• <!-- SOURCE: ..., qwen3-vl:8b @ 300dpi, N pages --> → high-confidence
• <!-- TIER: tesseract fallback --> → degraded; treat results conservatively

Step 3: Graceful handling of OCR noise

Hand-written math OCR will be imperfect. Expect:

• Greek letters misread as Latin ($\alpha \to a$, $\beta \to B$, $\pi \to T$ or $n$)
• Fractions rendered as flattened text ($\tfrac{dU}{dT} \to dUdT$ or similar)
• Subscripts/superscripts lost or inlined

Do not grade on algebraic correctness of OCR output. Instead, apply strategy-based grading:

Step 4: Strategy extraction from noisy MD

Read the converted MD file. For each problem, identify:

1. Which pattern(s) did the user invoke? Look for:

   • Named theorems / techniques the user wrote out ("Maxwell relation", "Stokes theorem", "by induction")
   • Variables they held fixed (even if notation is mangled)
   • Key intermediate objects (a chosen potential, a change of variable, an ansatz)

2. Did the reasoning reach the correct end form? Even if algebra is wrong, the user's final expression structure (Does it have a log? A sqrt? A series? Correct variables?) tells you if the approach worked.

3. Where did they stop? Incomplete work is common; note which step is the last recognizable one.

Step 5: Compare against reference solution

Open the reference (converted/solutions/<hw>.md for HW, quizzes/<name>_answers.md for quizzes, twins/<id>_<ts>_sol.md for twins, chain/<ts>_sol.md for chain).

For each problem/part, produce a verdict:

## P<n>
- Pattern match: ✅ / ⚠️ / ❌   [user invoked <Pk>, solution uses <Pk>]
- Variable choice: ✅ / ⚠️ / ❌  [user held <x> fixed, should be <y>]
- End form: ✅ / ⚠️ / ❌          [user's final: <form>, expected: <form>]
- Completeness: <last step user reached>
- Overall: <PASS | PARTIAL | FAIL>
- Note: <one line — what to study, which Pk to re-drill>

Do not report line-by-line algebra mistakes unless they are specifically about sign errors or notation bugs that matter on the exam (e.g., missing $-$ on $\kappa$ definition, conjugate vs. transpose confusion).

Step 6: Log errors

Canonical errors/log.md schema — single source of truth. Every command that appends here (/grade, /blind, future drills) MUST use exactly these keys. Downstream readers (statusline.py, weakmap, session_start.py) pattern-match on pattern: and problem_id: lines; any drift silently hides entries.

- problem_id: <id>
  pattern: <Pk>
  error_type: pattern-missed | wrong-variable | wrong-end-form | algebraic | sign | definition
  phase: reading | comprehension | transformation | execution | encoding   # optional (F2) — inferred from error_type when absent
  nature: slip | misconception | gap   # optional (F3) — inferred from error_type when absent
  summary: "<1 line>"
  source: answers/converted/<name>.md
  date: <ISO>
  overridden_by: <source>   # optional — present only on entries superseded by a human override

Only problem_id, pattern, error_type, summary, source, date are required (the six REQUIRED_KEYS log_tool.py enforces). phase, nature, and overridden_by are optional additive keys — omit them and downstream readers promote phase/nature from error_type via DEFAULT_PHASE/DEFAULT_NATURE (paideia_lib.iter_error_entries); write them explicitly only when the grading determined a phase/nature that differs from the inferred default.

Write through log_tool.py — never hand-edit the log. Build the YAML block for every non-✅ entry of this grading, then make ONE call:

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/log_tool.py" append \
  --source="answers/converted/<name>.md" <<'YAML'
- problem_id: <id>
  pattern: <Pk>
  error_type: <type>
  summary: "<1 line>"
  source: answers/converted/<name>.md
  date: <ISO>
YAML

The tool schema-validates every entry (keys, error_type values, date shape, source: equal to --source) and rejects the whole batch on any violation — fix the block and re-run rather than writing around it.

Why the tool exists — idempotent by source:, replace don't pile up. Re-grading the same answer (fix the OCR, re-run /grade) or re-running /blind on the same problem must NOT leave two copies of that attempt's errors in the log — the weakmap histogram would then double-count and over-rank those patterns. log_tool.py append deletes every existing entry whose source: equals --source before appending, atomically, so the log stays a record of the latest grading of each source, not a transcript of every re-grade. (A genuinely new attempt belongs under a new source: — a new upload gets a new filename, so this only collapses true re-grades of the same file.) If a grading produced zero errors on a re-grade, run log_tool.py remove --source=<source> so the stale entries clear.

Misgrade correction — origin-preserving override. When a user disputes a verdict (OCR misread, wrong error classification), use log_tool.py override instead of append. The original entries are preserved in the log with overridden_by: <source> injected after their date: line. The correction entries are appended as the new current verdict (no marker). This is the only path that maintains an audit trail of the original ruling.

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/log_tool.py" override \
  --source="answers/converted/<name>.md" <<'YAML'
- problem_id: <id>
  pattern: <Pk>
  error_type: <corrected-type>
  summary: "<corrected 1-line description>"
  source: answers/converted/<name>.md
  date: <ISO>
YAML

Rules: override when changing the verdict; remove only when the re-grade produced zero errors; never include overridden_by: in stdin (tool assigns it — passing it is rejected). The weakmap histogram counts only entries without overridden_by: as current verdicts; original-marked entries do not double-count. The six required keys and PATTERN_RX are unchanged — overridden_by is an optional additive key, not a seventh required key.

Step 7: Render grade summary (chat output)

Compact table, no verbose explanations:

| Problem | Pattern | Vars | End form | Overall |
|---|---|---|---|---|
| P1 | ✅ | ✅ | ⚠️ | PARTIAL |
| P2 | ❌ | — | — | FAIL |
| P3 | ✅ | ✅ | ✅ | PASS |

Dominant issue: pattern-missed on P2 (used brute-force integration; should use residue theorem, P7).
Drill next: /blind <problem testing P7>, or /pattern P7 for quick review.

Keep this under 15 lines of output.

Handling edge cases

Empty or unreadable PDF

If OCR yields <100 chars total, ask the user (in INTERFACE_LANG from .course-meta, default en): "OCR returned too little. PDF quality may be low or the handwriting too small. Options: (a) re-scan brighter/larger and re-upload (b) type the answer into .md and save it to answers/converted/<name>.md, then /grade again"

User uploads .md directly

Skip PDF conversion. Read answers/<name>.md directly. Everything else is the same.

Multi-page with disordered content

Hand-written work often has margin notes, arrows, struck-through attempts. OCR will render them chaotically. Note in the grade (in $INTERFACE_LANG): "Answer ordering ambiguous. My interpretation: . Let me know if different."

User already in context

If the user pastes their work directly into chat (not as PDF), grade it from context. Still apply strategy-based grading.

Anti-patterns (things NOT to do)

❌ Demand pixel-perfect algebra from OCR output ❌ Mark something wrong because OCR mangled a Greek letter ❌ Require the user to retype their solution in LaTeX ❌ Produce 3-page grade reports (stay compact) ❌ Reveal the reference solution before grading (user might be asking "did I get it right" as a first pass)

Integration

• Called by /grade
• Uses pdf skill for OCR
• Reads course-index/patterns.md (pattern IDs) and converted/solutions/ or equivalent
• Writes to errors/log.md and answers/converted/