Now let me read the key files.

Now the skills themselves and fact-check.md examples.

Now let's look at an actual fact-check.md file to get the concrete format.

Good, I have the fact-check format fully. Now let's look at the autoroad side.

Now the lint tool (fan-out lens/judge pattern), canon README, check_citations.py, and llm-tells.md.

Excellent material. Now let's check the tools/lint rules.json structure, notes/llm-tells.md, and plans/NEXT.md process sections, plus editing-pass skill for lens/judge fan-out.

Now let's check notes/llm-tells.md failure-mode content, plans/NEXT.md process sections, and pilot-report.md for the fan-out lens/judge pattern description and false-positive handling.

This is excellent data on the pilot report (fan-out lens/judge, false positive rate, promotion criteria). Now let me check plans/NEXT.md process sections and farming-check skill quickly for priority ordering references, then write the report.

Report: Fact-checking («OTHER-PROJECT») and consistency-checking (autoroad) methodologies

1. «OTHER-PROJECT»: fact-checking

Where it lives. redacted: private sets the top-level rule: "Factually accurate and fact-checked before the draft is called done" (goal 2), plus a house rule: "Never invent quotes, benchmarks, or output; anything presented as program output must have actually been produced by a program, with the receipt stored in the post's directory." Pipeline order in CLAUDE.md: brief → research (post-research skill) → experiments → draft → **fact-check** (fact-check skill) → voice pass (voice-pass` skill) → hand off.

Receipts and the "ran it myself" vs "quoted from source" distinction. From .claude/skills/post-research/SKILL.md step 2: every research finding needs "a receipt: file:line, commit SHA, URL, or quoted output — and each marked "ran it myself" vs "quoted from source"." Rule: "No receipt, no claim." Research files are explicitly not trusted as ground truth — step 3 ("Verify before you trust") states "Research reports are drafts of the truth, not the truth," with a concrete example: a research file claimed PR #424 added Phase::Explain, but the source (src/runner.rs) said "not yet supported." When a research file is found wrong, it's corrected in place with a CORRECTION: annotation, never silently rewritten — "the record of what was wrong and how it was caught is part of the receipts."

Claim-by-claim format. .claude/skills/fact-check/SKILL.md and the worked example posts/pretty-printing/fact-check.md (54 claims, 4 errors) define it concretely:

• A fresh subagent (sonnet) with the draft, research/ and experiments/, and access to primary sources (repo checkouts, URLs), but not the drafting conversation — "familiarity with how a claim was written is exactly what makes errors invisible."
• The agent enumerates every checkable claim (numbers, names, dates, quoted code, quoted output, API behaviour, history) and checks each against the primary source, not the research report.
• Table format actually used, columns: # | Claim | Verification | Verdict, e.g. row 15: "...roughly two thousand lines of implementation..." | git diff --numstat of the actual merge: ~5,968 non-test lines, ~5,115 test lines | WRONG — see Problems #1.
• Verdict categories observed: CONFIRMED, CONFIRMED (behaviorally accurate; ...), CONFIRMED (abbreviated but accurate), WRONG, OVERSTATED / ambiguous, and a "Minor formatting difference" tier. The skill text also names unverifiable as a category.
• A "Problems found" section up top lists only the substantive errors with full reasoning, followed by the full 54-row claim log — so the file reads as "settled" once fixed: a > **Resolution (date):** block at the very top of fact-check.md records exactly what was changed to fix each numbered problem.

Claim classes to be paranoid about (fact-check/SKILL.md step 3, drawn from all four errors in post 1): numbers/counts (miscounted lines, comparison direction reversed); quoted code (must be byte-verbatim against the pinned commit — a format! block had been "lightly paraphrased," reflowed but not reworded); absolute statements ("DateTime doesn't have a Debug representation" — false as literally written; chrono's Debug output just isn't valid Rust syntax); distributional claims ("for almost every drawn value" asserts an unmeasured distribution — "Only assert what was actually measured").

Fanning research to small models, then synthesis. post-research/SKILL.md step 2: "One subagent per question, dispatched in parallel: haiku — fetching and summarising public pages, docs, changelogs. sonnet — reading code, git history/archaeology, design or style judgement." Synthesis happens before drafting (outline step) and again explicitly in fact-check step 7 of the same skill ("Synthesize before drafting").

Revision after fact-check. fact-check/SKILL.md step 5: "Wrong claims get corrected; unverifiable ones get accurately hedged, cut, or turned into an explicit question for David — never smoothed over." Step 6: "David's edits are in scope" — the pass is re-run after his rewrites too, "the same as after mine." Step 7: commit the pass (fact-check.md + draft fixes) as its own commit — house rule of one state per commit.

Voice/LLM-tell riders. Fact-check and voice-pass are separate skills but run in the same pipeline position (both "before draft is called done," both "repeat after substantial revision"). voice-pass/SKILL.md runs a mechanical lint (voice_lint.py), a stylometric Claude-vs-human classifier trained on David's corpus (anchors: his prose ~6-8% "Claude," real Claude prose ~93%, threshold ~30%), then three blind parallel model passes with no access to each other or to revision history: Pass A voice fidelity, Pass B LLMism device hunt (named devices in notes/voice.md §7), Pass C coherency/clarity. A closed feedback loop is specified: whenever David flags a new tell, (1) fix the instance, (2) add a named device entry to notes/voice.md §7, (3) if mechanically detectable, add a lint pattern to voice_lint.py — "The catalogue is the asset; the passes are only as good as it is."

2. autoroad: consistency-checking

Skill separation. consistency-check verifies a draft against existing canon; consistency-backfill grows canon from a chapter's implications (a "two-way negotiation"). Pipeline per notes/canon/README.md/plans/NEXT.md: draft → flat-restatement pass → canon lens (sonnet, facts) → pre-David pass (opus, advisory, "never cuts a reason") → commit → author reads → comments applied. consistency-check/SKILL.md states the same ordering explicitly: "consistency-check → editing-pass → consistency-backfill."

Claim levels (notes/canon/README.md, mirrored in consistency-backfill/SKILL.md): Established (on-page in an accepted chapter, cited), Working (binding until overruled — design decisions and Claude proposals, attributed), Open questions — as of a 2026-08-28 triage, no canon entry may sit as bare "open": each is answered, ruled, or deferred with a named trigger (tracked centrally in deferred-questions.md). Rule shared by both skills: "never delete or weaken an existing canon entry to accommodate a draft — that decision belongs to the author; flag it as a contradiction instead."

Priority ordering of checks in consistency-check/SKILL.md. Ground truth is read in explicit priority order: (1) notes/canon/**/*.md (system, characters, world, deferred-beats — hardest facts), (2) notes/concept.md/premise.md (design intent), (3) notes/theme.md ("softer than canon: a draft contradicting the theme is a flag, not automatically an error... a draft foreclosing a theme commitment... is a serious finding"), (4) notes/maybe-later.md, then previously accepted chapters in order. "If canon and an earlier chapter conflict, canon wins; flag the earlier chapter too." What-to-check categories are ordered System mechanics → character continuity/knowledge-state → world facts → timeline → physical/scene continuity → voice-level facts → "affordances have infrastructure" — i.e., hard mechanical/numeric facts first, softer drift-prone checks (theme, register) last.

Finding schema (consistency-check/SKILL.md, "Reporting"): most severe first; for each finding — Quote the offending line(s) with file/location; Cite the conflicting canon (file + canonical statement) or prior-chapter line; Classify: contradiction (violates canon), drift (technically compatible but strains established facts), ambiguity (could be read as violating; suggest disambiguation); Suggest the minimal fix without applying it — "silent edits to prose are never acceptable in this pass." "No findings is a valid result; say so plainly rather than inventing nits."

Contradiction vs drift, applied. The distinction is explicitly a severity/certainty split, not a topic split: a numeric or on-page violation is contradiction; something that technically fits but stretches plausibility (e.g., an affordance not backed by established infrastructure) is drift; something readable either way is ambiguity.

Backfill mechanics (consistency-backfill/SKILL.md): extract every implication of a scene by category (character history, geography/travel arithmetic, institutions, economy, magic, timeline/season, demography — "Include implications of absence: nobody reacts to X ⇒ X is normal here"), then negotiate: fits canon → record; canon silent → commit to smallest Working addition ("prefer under-specification"); conflicts → propose the cheaper fix (chapter line vs Working canon revision), with Established canon changes and anything with dependents escalated to the author as a flagged decision, never auto-applied. A separate "corner-watch" step flags anything that will constrain future books (hard numbers, absolute statements, world-scale facts) for author sign-off even when internally consistent — "the question is not 'is it consistent' but 'do we want to be stuck with it.'"

The lens/judge fan-out (tools/lint/). rules.json holds 110 rules across three tiers — regex (38, cheap deterministic pattern match, e.g. negative-parallelism: \bnot\s+[^,.;:!?]{2,45},\s*but\b), metric (13, budget/count-based, e.g. em-dash rate, aphorism count per scene), and semantic (59, needs judgement, tagged with a lens, e.g. generalising-clause-budget under lens: "narration"). Each rule carries severity, why, source (traced to a specific ruling/comment), examples, counter_examples, and (for regex) an allowed list of pre-cleared matches. rules-notes.md explicitly sorts rules into "Best suited for haiku-level detection" (surface-visible, decidable from a sentence alone), "Workable with care" (needs local context), and "Poorly suited to haiku — needs canon, the whole draft, or taste; route to opus or to a deterministic tool" — e.g. canon-scale-contradiction and system-terminology need notes/canon/**, contamination-overlap should never go to a model at all ("use tools/overlap_check.py, never a model").

editing-pass/SKILL.md implements the fan-out concretely: "Pass 0: mechanical pre-pass" — deterministic grep for month-name errors, banned lexicon, em-dash budget vs prose-only word count, negative-parallelism shapes, name-spelling variants, word-frequency census — added after "the verifier benchmark showed even good model verifiers miss greppable defects." Then Pass 1 (sense check: inverted intent, broken logic, reference failures, impossible staging, wrong-way comparisons, dialogue drift) and Pass 2 (voice compliance against notes/voices.md/notes/llm-tells.md). Findings are reported "sense errors are always fix-worthy; voice findings are recommendations the author may overrule." From draft/book-01/VOICE.md §3.4, quoted in rules-notes.md: "lens findings are advisory, and a finding that cuts an explanation, a reason, an answer or a qualification is presumptively wrong. The pipeline should surface findings, never auto-apply them."

False-positive avoidance, concretely measured. tools/lint/pilot-report.md: a pilot of 19 machine-flagged candidates over 4 files gave 8 confirmed, 2 borderline, 9 false-positive (47% FP overall). Per-rule breakdown showed contamination-overlap and description-list at 0% FP, while canon-scale-contradiction, announce-before-deliver, cause-left-out, implication-stopped-short were 100% FP — diagnosed as one shared cause: "the scanner reads a quote without its neighbouring sentence." The rule promoted to a hard deterministic gate (contamination-overlap, exact n-gram matching against a corpus of deleted drafts) is the one with a 0% judged FP rate; rules staying "judged" (semantic tier) get counter-examples added directly into rules.json to suppress the specific FP pattern found. Conclusion stated verbatim: "cheap wide hunting plus expensive narrow judging caught real contamination and a real budget overspend... at a cost of nine cheap rejections... With those [fixes], the projected FP rate on this sample falls from 47% to roughly 10%."

Citation/quote verification tool. scripts/check_citations.py — added "after a hallucinated quotation... was found cited as Established to ch. 021 in four canon files while appearing in no chapter at all. It survived a whole-story consistency audit... A soft check did not catch it; this is the mechanical one." It regex-extracts "quoted text" (ch. NNN) patterns from notes/, plans/, summaries/, normalizes whitespace/quotes/dashes, and checks the quote actually appears (verbatim or per-sentence, for tag-broken quotes) in the cited chapter file. Buckets findings into FABRICATED (no substantial match anywhere), ALTERED (real quote, extended or reworded), WRONG_CHAPTER (verbatim but cited to the wrong chapter). The docstring documents a known blind spot (citation-order ambiguity causing ~7-8 spurious findings) and records that two attempted fixes made results worse — kept as a documented limitation rather than "fixed" incorrectly, with the note "If you try again, do it with a test corpus, not by eye."

3. Stated failure modes of LLM fact/consistency-checkers themselves

• Fabricated citations/quotes (autoroad): the check_citations.py docstring's origin story is exactly this — an LLM-authored canon file cited a quotation as "Established... (ch. 021)" that appears nowhere in the text, and it "survived a whole-story consistency audit" because the audit was itself a soft (model) check rather than a mechanical one.
• Confirmation bias / familiarity blindness («OTHER-PROJECT»): fact-check/SKILL.md step 1 requires the checking agent be denied "the drafting conversation" because "familiarity with how a claim was written is exactly what makes errors invisible." Similarly voice-pass/SKILL.md requires the three review passes be "blind" — no earlier reviews, no revision history, no knowledge of who wrote which sentence — because "fresh eyes catch what the author-agent has gone blind to; this is load-tested, not theory."
• Trusting your own research as ground truth («OTHER-PROJECT»): "Research reports are drafts of the truth, not the truth" — checking must go to primary sources, not to the research/summary layer, because that layer itself can be (and was) wrong.
• Over-flagging / narrow-context false positives (autoroad): the pilot report's dominant failure mode was a cheap scanner reading a quote in isolation and missing an adjacent sentence that resolved it, producing 100%-FP rules; the fix applied was adding context windows and counter-examples, not discarding the fan-out architecture.
• Model verifiers miss "greppable" defects (autoroad): `editing-pass/SKILL.md's Pass 0 was added specifically because "the verifier benchmark showed even good model verifiers miss greppable defects" — i.e., LLM judges are worse than deterministic tools at exactly the class of error a regex or exact-match script would catch reliably, which is why the architecture insists on mechanical gates before judgment passes wherever a mechanical gate is possible (contamination-overlap, citation-checking, em-dash counts, banned lexicon).
• Smoothing over uncertainty («OTHER-PROJECT»): explicit house rule — "When uncertain about a fact or a framing, ask David rather than smoothing over it" — framed as a failure mode to actively resist (an unverifiable claim must be hedged, cut, or escalated, never quietly asserted).