comfy-corpus-validation

comfy-corpus-validation

comfyui-sampler-info ships a JSON corpus (web/data/samplers.json, web/data/schedulers.json, web/data/models.json) of human-written facts about ComfyUI's sampler and scheduler tokens — year, family, ODE order, summary, good_for, pairs_with, and per-model default recipes. The pack's entire value is that those facts are true. A wrong fact is worse than a missing one: the UI presents it with the same confidence either way.

This skill is the method for making corpus claims verifiable instead of remembered. It exists because three factual misses shipped or nearly shipped in one day, each caught by luck: a blog-sourced sampler recommendation that the vendor's own template contradicted, a schedule-behaviour claim written from model memory that computing the sigmas falsified, and a model family absent from the corpus entirely because nothing ever asked what the live install offers that we describe. The first real run of the tooling below immediately found a fourth nobody knew about.

The core rule

A claim about ComfyUI is only as good as the highest rung you verified it on. Never write a Tier-3 claim that a Tier-1/2 source could settle but didn't.

When to Use This Skill

Use this skill when...	Use instead when...
Adding or editing an entry in web/data/*.json (sampler, scheduler, model recipe)	Reading generation settings out of an output PNG/WebP/MP4 -> comfy-metadata
Auditing whether the corpus's existing facts are still true	Checking whether the pack itself works in a live install -> comfyui-pack-live-smoke
A source claims "model M wants sampler S" and you must decide whether to believe it	Debugging the pack's publish/release pipeline -> comfy-registry-lifecycle
Asking what the install offers that the corpus says nothing about	Writing the pack's frontend/backend code -> comfyui-node-authoring

comfyui-pack-live-smoke verifies the pack functions. This skill verifies its facts are true. A pack can be perfectly working and comprehensively wrong.

The source-of-truth ladder

Tier	Source	Settles
1. Executable ground truth	/object_info/<NodeClass> on a live install (token lists + python_module); comfy/samplers.py (SCHEDULER_HANDLERS, KSAMPLER_NAMES, SAMPLER_NAMES); custom-node source; computing the sigma curve by calling the real handler	Does the token exist? Who provides it (core vs which pack)? What does the schedule actually do?
2. Shipped vendor artifacts	comfyui_workflow_templates_json/templates/*.json — the KSampler widget values in the official template are the vendor's default recipe; docs.comfy.org; the HF model card	What does the vendor actually recommend for model M?
3. Secondary sources	Blogs, tutorials, Civitai, Reddit	Discovering claims. Never the sole basis for one.
4. Empirical testing	Your own generation results	Legitimate — but must be labelled (source.kind: "empirical"), and where possible backed by a Tier-1 mechanism.

Tier 1 beats Tier 2 beats Tier 3. Tier 4 is not a lower Tier 3 — it is a different kind of claim (yours, not someone's), and honest labelling is what keeps it usable.

The three protocols

1. Measure behaviour, never assert it

Any claim about what a schedule does — "concentrates steps at the low-noise end", "front-loads denoising", "spends steps in the mid-range" — is computed, not remembered. Call the real handler against a real ModelSampling, get the sigma list, bucket the steps into noise bands, and read the answer off the table.

This is the only thing that caught the beta57 error: the corpus claimed it "spends steps in the mid-range rather than at the extremes". Measurement showed it does the exact opposite — it takes steps out of the mid range and spends them at the low-noise end. No amount of plausible reasoning would have found that; one computation did.

Reading the source is not the same as executing it. The first real run of corpus-check caught a fourth error: linear_quadratic's prose was written by reading comfy/samplers.py — correctly — but missing the 1.0 - x inversion one line down, so the claim came out exactly backwards (it spends 17 of 20 steps at high noise, not low). Source-reading is Tier 1 and still produced a confident falsehood. Only running the function and looking at the numbers it returns catches this class. Full numbers in REFERENCE.md.

2. Disagreement means escalate or stay silent

When two Tier-3 sources conflict, resolve at Tier 1/2, or do not write the claim. Never break the tie by picking the better-ranked blog — search ranking is not evidence. A top-ranked guide said Krea 2 wants er_sde; ComfyUI's own shipped workflow template says euler. The template is the vendor's recipe and it wins. When no Tier-1/2 source can settle it, the correct corpus entry is no entry — the UI's (no metadata for this option yet) is honest; a confident wrong sentence is not.

3. Coverage is a first-class check

Ask, every time: "what does the live install offer that the corpus says nothing about?" Krea 2 was missing from the corpus entirely, and nothing in the process ever asked the question, so the gap stayed invisible until a user did. An undescribed token is a visible hole in the pack's UI. Coverage is not a nice-to-have audit; it is one of the three checks corpus-check runs.

Two traps that will bite you

Template samplers hide inside subgraphs. In image_krea2_turbo_t2i.json and flux1_krea_dev.json the KSampler node lives under definitions.subgraphs[].nodes, not top-level nodes. A naive top-level scan returns nothing, which reads as "this template configures no sampler" — exactly backwards, since the template is the authoritative answer. Always walk definitions.subgraphs[] too.

A field that looks like the answer isn't. The Comfy Registry API's latest_version is not the newest version. Same class of error: a plausibly-named field is not evidence. Check what the field means before trusting what it says.

Running the check

Ground truth is regenerated on demand, never committed — a committed snapshot is just another stale secondary source.

COMFY_SSH_HOST=popos.intra.lakuz.com just corpus-check

Two halves, split because only one of them needs a GPU box:

• scripts/corpus_probe.py runs on a ComfyUI host (needs torch + ComfyUI importable). It emits ground truth as JSON on stdout:
  • tokens — sampler/scheduler tokens the live install offers, each tagged with its provider (core vs which custom-node pack).
  • schedulers — the measured sigma curve per scheduler plus the step allocation across low/mid/high noise bands.
  • templates — {template, sampler, scheduler, steps, cfg} rows from the shipped workflow templates, including subgraph-nested KSamplers.
• scripts/corpus_check.py runs locally, stdlib-only. It consumes the probe JSON plus web/data/*.json and emits a structured === SECTION === / KEY=VALUE / STATUS= report.

Reading the report

Section	What it means	What to do about it
COVERAGE	Tokens the install offers that the corpus doesn't describe (resolved exact → alias → prefix)	Write the entry, or accept the hole deliberately
RECIPES	Every models.json entry citing source.template re-checked against the probe's template rows	A FAIL means upstream changed the template — that is the point, not a bug. Re-read the template and update the recipe
BEHAVIOUR	The measured band table shown beside any entry whose prose makes a step-allocation claim	Read the prose against the numbers. If they disagree, the prose is wrong

A RECIPES FAIL is the check earning its keep: it means the vendor moved and the corpus didn't.

Labelling provenance

Any corpus entry may carry a source field. kind is one of:

kind	Meaning
vendor-default	From a shipped workflow template — the only kind carrying a machine-checkable template, and the only one corpus-check can re-verify
vendor-doc	docs.comfy.org or the model card
pack-provided	Documented by the custom-node pack that ships the token
community	Tier 3. Use sparingly, and never for a claim Tier 1/2 could settle
empirical	Your own testing. Honest, and must say so
paper	The originating paper

models.json stores each model family's vendor-default recipe exactly once, with the template it came from — so the recipe has one home and one checkable provenance, rather than being restated in prose across several entries where it can drift.

See also

REFERENCE.md — the four misses worked end to end with the real numbers: the er_sde disagreement and its Tier-2 resolution, the beta57 sigma measurement that falsified our own prose, the linear_quadratic inversion that survived a correct reading of the source (and had already propagated into another entry's reasoning), the subgraph-hidden KSampler and its exact widgets_values vector, and how a Tier-1 /object_info diff mechanically proves which pack provides which scheduler.