From write
Quality gate measuring prose flatness via local perplexity computation using Ollama + Ministral 8B. Returns pass/fail verdict with diagnostics for automated workflow gates.
npx claudepluginhub pknull/asha-marketplace --plugin writeThis skill uses the workspace's default tool permissions.
Automated quality checkpoint for prose using local perplexity measurement. Runs Ministral 8B via Ollama to compute token-level perplexity, detecting "flat" AI-generated prose patterns. Acts as a gate in chapter creation workflows.
Creates isolated Git worktrees for feature branches with prioritized directory selection, gitignore safety checks, auto project setup for Node/Python/Rust/Go, and baseline verification.
Executes implementation plans in current session by dispatching fresh subagents per independent task, with two-stage reviews: spec compliance then code quality.
Dispatches parallel agents to independently tackle 2+ tasks like separate test failures or subsystems without shared state or dependencies.
Automated quality checkpoint for prose using local perplexity measurement. Runs Ministral 8B via Ollama to compute token-level perplexity, detecting "flat" AI-generated prose patterns. Acts as a gate in chapter creation workflows.
ollama serve)ollama pull mistral)requests libraryBased on Claude Book research:
| Metric | PASS | WARNING | FAIL |
|---|---|---|---|
| Sentence PPL < 22 | <10% of sentences | 10-20% | >20% |
| Low variance window | 0 spans | 1-2 spans | 3+ spans |
| Consecutive low-PPL | <3 sentences | 3 sentences | 4+ sentences |
| Overall low-PPL ratio | <20% | 20-30% | >30% below PPL 25 |
Key insight: Low perplexity = predictable tokens = "flat" prose. High perplexity = surprising tokens = textured prose.
Prose text to evaluate (minimum 250 words for reliable measurement).
verdict: PASS | WARNING | FAIL
metrics:
mean_perplexity: 34.2
median_perplexity: 31.8
variance: 12.4
low_ppl_ratio: 0.18 # % sentences with PPL < 25
consecutive_low_max: 2 # longest run of PPL < 22
low_variance_windows: 1 # spans with uniform PPL
flagged_sentences:
- sentence: "She felt a sense of profound realization wash over her."
perplexity: 18.3
position: 7
- sentence: "The morning light filtered softly through the curtains."
perplexity: 16.9
position: 12
recommendation: PROCEED | REVISE_WITH_VS_TAIL | ESCALATE_TO_HUMAN
rewrite_guidance: |
[Only present if verdict = FAIL]
Flagged patterns:
- 4 consecutive low-PPL sentences (positions 7-10)
- 28% of text below PPL 25 threshold
- Low variance window detected (positions 15-22)
Suggested intervention: VS-Tail sampling
Target positions: 7-10, 15-22
# Check a file
python scripts/check_perplexity.py path/to/chapter.md
# Check with custom threshold
python scripts/check_perplexity.py path/to/chapter.md --threshold 20
# JSON output for integration
python scripts/check_perplexity.py path/to/chapter.md --json
Use the perplexity-gate skill to evaluate this prose:
The morning light filtered through the curtains, casting long shadows across
the hardwood floor. Sarah stretched lazily, feeling the warmth of the sun on
her face. She knew today would be different. Something in the air told her
that change was coming, a subtle shift in the atmosphere that she couldn't
quite put into words.
After Draft 1 completion, run perplexity-gate:
- If PASS: proceed to developmental-editor
- If FAIL: apply VS-Tail sampling to flagged sections, then re-check
- Max 3 rewrite iterations before human escalation
When verdict is FAIL:
asha/modules/verbalized-sampling.mdNine methods for improving perplexity:
| Technique | Description | When to Use |
|---|---|---|
| Verbalized Sampling | Generate N variants with probability estimates, select p < 0.10 | Default first attempt |
| Fragmentation | Break long sentences, vary length | Metronomic rhythm detected |
| Character Voice | Inject character-specific speech patterns | Generic dialogue |
| Rare Vocabulary | Substitute common words with less frequent synonyms | High word-level predictability |
| Syntactic Inversion | Rearrange clause order | Repetitive structure |
| Sensory Details | Add specific, unusual sensory observations | Generic descriptions |
| Broken Rhythm | Intentionally disrupt sentence flow | Uniform cadence |
| Cliche Subversion | Twist expected phrases | Predictable metaphors |
| Narrative Ellipsis | Remove expected information, imply instead | Over-explanation |
| Agent | Relationship |
|---|---|
| ai-detector | Deprecated for this use case. Use perplexity-gate for local measurement |
| prose-analysis | Complementary. prose-analysis = subjective craft; perplexity-gate = objective metrics |
| consistency-checker | Sequential. Consistency runs first, perplexity-gate runs after draft generation |
What this measures:
What this does NOT measure:
Important: A PASS verdict means the prose has sufficient variance in predictability. It does NOT mean the prose is good. Always run prose-analysis for craft evaluation.
For chapter-creation.yaml:
gates:
- name: perplexity
description: "Prose has sufficient token-level variance"
blocking: true
max_retries: 3
on_max_retries: escalate_to_user
bypass: "Human override with justification"
Symptom: Script errors with connection refused.
Solution: Ensure Ollama is running (ollama serve) and model is pulled (ollama pull mistral).
Symptom: Every draft fails regardless of content quality.
Causes:
Solution: Adjust PPL thresholds for project, or bypass with human override.
Symptom: Obviously AI-generated content passes.
Causes:
Solution: Ensure minimum sample size, tighten thresholds.
Symptom: Takes >30 seconds per check.
Solution:
--sample flag to check subset of sentences| File | Purpose |
|---|---|
SKILL.md | This documentation |
scripts/check_perplexity.py | Main perplexity computation script |
asha/modules/verbalized-sampling.md — Rewrite technique referencewrite/agents/prose-analysis.md — Craft quality evaluationwrite/skills/novel-state/ — State management for metrics tracking