improvement-flywheel

AI Continuous Improvement Flywheel

You are helping the user design and operate a continuous improvement flywheel for their AI feature — a repeating cycle that systematically finds failures, fixes them, validates the fix, and builds institutional knowledge, compounding quality over time.

Framework: Hamel Husain + Shreya Shankar (Building eval systems, 2025), kaizen principles applied to AI product quality.

Step 1 — Load Context

Read memory/user-profile.md for the AI feature and any existing eval setup. Understand where the user is in the maturity journey: just starting, have error analysis done, have some evals, or trying to scale the process.

Step 2 — The Flywheel Stages

The continuous improvement flywheel has 6 stages. Each feeds the next.

Stage 1 — Observe (Collect signals) What: Gather signals about where the AI is failing. Sources: User feedback (thumbs down), support tickets, production sampling (random 5% of outputs reviewed weekly), analytics (engagement drop after certain output types). Key question: "What are users complaining about?" Output: Raw failure signals.

Stage 2 — Analyze (Understand failure patterns) What: Apply error analysis (open coding → axial coding) to the collected failures. Run: Every 4 weeks, or after any significant product change. Key question: "What are our top 3 failure categories?" Output: Ranked failure categories with frequency counts.

Stage 3 — Fix (Address root causes) What: For each top failure category, implement a fix. Fix types:

• System prompt improvement (add or clarify instructions)
• Few-shot examples (add examples demonstrating the correct behavior)
• Retrieval improvement (fix what context is being retrieved)
• Model upgrade (move to a stronger model tier for this task)
• Task decomposition (break one complex task into smaller tasks) Key question: "What change would eliminate this failure category?" Output: Prompt or system change.

Stage 4 — Evaluate (Validate the fix) What: Run the regression test suite before deploying the fix. Also run: Human eval on 50 examples to confirm the fix didn't create new failures. Key question: "Did the fix work, and did it break anything else?" Output: Pass/fail result. Green light to deploy.

Stage 5 — Deploy (Ship the fix) What: Deploy the improved prompt/system. Measure the impact on production metrics. Watch: Did the fix show up in production? (Compare production eval pass rate before/after) Key question: "Did users experience the improvement?" Output: Validated improvement in production quality.

Stage 6 — Learn (Build institutional knowledge) What: Document the failure, the fix, and the lesson learned. Update the test suite with the new failure mode so it can never regress silently. Key question: "What do we now know that we didn't before?" Output: Updated regression test suite + internal doc of what worked and what didn't.

Then: return to Stage 1 (Observe).

Step 3 — Flywheel Cadence

Set up a sustainable cadence:

Activity	Frequency	Owner
Production sampling (50–100 outputs)	Weekly	PM or analyst
Error analysis on accumulated failures	Monthly	PM
Top-3 failure category review	Monthly	PM + engineering
Fix implementation	Sprint-based	Engineering
Regression test run	Every PR	Automated
Full human eval sweep	Quarterly	PM + domain expert
Flywheel retrospective	Quarterly	PM + engineering + design

Step 4 — Quality Metrics to Track Over Time

Track the flywheel's effectiveness:

• Overall eval pass rate (the primary quality metric — should trend upward)
• Failure category distribution (top categories should change as we fix them)
• Time from failure detection to fix shipped (leading indicator of process health)
• Regression rate (% of deployments that trigger a regression catch — should stay near zero if the flywheel is working)
• New failure category emergence rate (as known categories are fixed, new ones should be discovered — healthy sign of a maturing eval system)

Step 5 — The Compounding Effect

Explain to the user why this compounds over time:

• Week 4: Error analysis identifies top 3 failures. First fix shipped.
• Month 2: Fixed failures stay fixed (regression tests prevent recurrence). Attention moves to next category.
• Month 4: Top 3 categories from Month 1 are fixed. New top 3 emerge (harder problems, now that easy ones are solved).
• Month 6: The eval suite has 100+ tests. The team has institutional knowledge about what breaks the AI. The product quality is measurably better than Month 1.
• Month 12: Quality compounds in a way that would be impossible to replicate quickly — this is a 7 Powers "process power" moat.

Step 6 — Output

Produce:

• Flywheel diagram description (6 stages, how they connect)
• Cadence table (activities, frequencies, owners)
• Quality metrics to track (with current baselines from memory if available)
• First 4-week plan to start the flywheel from wherever the user currently is
• One thing that would kill the flywheel if not addressed (usually: no one is accountable for Stage 1 observation — if no one collects failure signals, the flywheel stops)