Document extraction — taguru extract

The CLI shape

taguru extract --context sake --description "酒蔵の知識" \
  --out batches/ docs/                # every .md/.txt under docs/, in name order
taguru import batches/               # offline — or POST each file to /import

PDFs are not among the formats above — nor are HTML, DOCX, PPTX, or objects in S3-compatible storage. The standard ingest connectors (Python SDK) read all of those into the same batch contract this page describes; the local RAG walkthrough shows one worked end to end.

TAGURU_EXTRACT_URL      OpenAI-compatible /chat/completions endpoint (required)
TAGURU_EXTRACT_MODEL    model name (required)
TAGURU_EXTRACT_API_KEY  Bearer credential (optional)
TAGURU_EXTRACT_SCHEMA   default for --schema (default unset)
TAGURU_EXTRACT_TIMEOUT_SECS  time budget per completion; 0 = unlimited (default 300)
TAGURU_EXTRACT_PARALLEL  concurrent chunk completions per document (default 1)
TAGURU_EXTRACT_FACT_BUDGET  default for --fact-budget (default 0, off)
TAGURU_EXTRACT_MAX_ATTEMPTS  total attempts at valid JSON per chunk, 1-10 (default 2)
TAGURU_EXTRACT_CORRECTIVE_CONTEXT_BYTES  cap a corrective turn's replay of
                    the model's own prior bad answer to this many bytes;
                    0 omits it entirely (default: unset, replay in full)
TAGURU_EXTRACT_STRUCTURED_OUTPUT  default for --structured-output (default off)
TAGURU_EXTRACT_MAX_OUTPUT_TOKENS  default for --max-output-tokens (default unset)
TAGURU_EXTRACT_ESCALATION_FACTOR  cap of the one escalated resend after an
                    answer ends at --max-output-tokens, as a multiple of that
                    budget; 0 = uncapped (default 2)
TAGURU_EXTRACT_CHUNK_BYTES  default for --chunk-bytes (default 24576)
TAGURU_EXTRACT_TRACE_ATTEMPTS  `off` disables the per-document attempts log (on by default; see Trace)
TAGURU_EXTRACT_LOSSY  default for --lossy (default 0/false)
TAGURU_EXTRACT_CANDIDATES  default for --candidates (default 0/false)
TAGURU_EXTRACT_VOCABULARY  default for --vocabulary (default unset, off)
TAGURU_EXTRACT_COVERAGE  default for --coverage (default 0/false)
TAGURU_EXTRACT_DIAGNOSTICS  default for --diagnostics-out (default unset, off)
TAGURU_EXTRACT_DIAGNOSTICS_RAW_BYTES  attach the model's raw answer text to
                    each diagnostics record, capped to this many bytes;
                    default unset/0 = never attach it (metadata only)

--dry-run           list what would extract or skip; calls nothing
--force             re-extract documents the manifest says are unchanged
--no-passage        omit the document body from the batch (facts only)
--questions N       doc2query: also propose up to N retrieval questions
                    per paragraph (1..=8; incompatible with --no-passage)
--fact-budget N     ask the model to keep each chunk's answer to at most N
                    associations total (default 0, off); a soft instruction,
                    never enforced after the fact — overrides
                    TAGURU_EXTRACT_FACT_BUDGET when both are set
--structured-output MODE  constrain the answer's shape on the wire
                    (see below): auto | json-schema | json-object | off
                    (default off — today's plain request)
--max-output-tokens N  explicit output budget per completion, sent as
                    max_tokens (default: none sent); an answer cut off at
                    the budget escalates deterministically instead of being
                    re-asked under the limit it just hit (see below)
--chunk-bytes N     document bytes per model call (default 24576, at least
                    512); chunks split at paragraph boundaries — lower it for
                    a slow provider or output-dense documents (statutes,
                    minutes); overrides TAGURU_EXTRACT_CHUNK_BYTES
--parallel N        chunk completions to run concurrently within one
                    document (default 1, sequential); overrides
                    TAGURU_EXTRACT_PARALLEL when both are set
--lossy             restore the pre-#199 behavior: drop a business-rule-invalid
                    item (bad weight, dangling alias, out-of-range question, …)
                    and count it instead of correcting or failing (see below)
--candidates        offer the document's own names (kanji/katakana compounds,
                    ASCII identifiers — segmented deterministically, no
                    dictionary) as preferred subject/object spellings;
                    non-restrictive, off by default; toggling re-extracts
                    (ADR 0014)
--vocabulary PATH   steer spellings toward a target context's existing
                    vocabulary: PATH is an exported batch stream or a
                    directory of them (taguru export --out DIR); concept
                    names and labels are offered as preferred spellings,
                    and a context spelling never fails the occurrence
                    check; off by default, content-digest re-extracts
                    (ADR 0015)
--coverage          report every sentence holding two or more of the
                    document's own names (the --candidates segmentation)
                    that no extracted association covers — one stderr line
                    per sentence, a count on the report line; report-only,
                    off by default, never re-extracts (ADR 0016)
--diagnostics-out FILE  write a JSONL sidecar of tagged records (see below):
                    one per chunk (provenance), one per LLM attempt (source,
                    chunk, attempt, terminal state, finish_reason, token
                    usage, latency, parse/validation issues), one per
                    document written (association/alias/duplicate/dropped/
                    uncovered counts), a "run" record first; off by default,
                    ignored under --dry-run. Joins the per-document trace
                    OUT/.extract-trace/ always writes (see below)
--source-id ID      write ID as the batch header's source instead of the
                    document path — the promotion runbook's
                    session:{agent}:{id} convention; several documents each
                    get ID/{file stem}, and a collision fails (import
                    retracts-then-applies per source id). Empty or
                    whitespace-only IDs are refused, and IDs over 1024
                    bytes (the server's source-name cap) fail up front
                    rather than at import. Changing it rewrites the batch
                    but reuses cached chunk answers (ADR 0017)
--date WHEN         the session's own date, written on the passage line:
                    YYYY-MM-DD (UTC midnight) or positive epoch seconds;
                    needs the passage
--tag TAG           tag the batch's source (repeatable, deduplicated),
                    written on the passage line; needs the passage
--context NAME      the context every batch file targets
--description TEXT  attach a create block (used only when the context is absent)
--schema FILE       a context schema document (see below) — the same shape
                    {stem}.schema.json/PUT /contexts/{name}/schema persist and
                    serve; a file that fails to parse or fails schema::install's
                    own checks is a startup error, never a silent skip
--config F          read KEY=VALUE environment from F (same dialect as serve)

Exit codes: 0 every document extracted or skipped · 1 some documents failed (details on stderr; the rest completed) · 2 usage error.

--schema is this pipeline's own copy of a context schema document — not to be confused with --structured-output json-schema below, which is an unrelated OpenAI wire format for the model's answer shape. Extract is offline and has no running server to fetch one from, so unlike the LangChain SDK ingesters (which pull it live), the operator hands it the file explicitly.

The credential boundary survives

The server holds no model credentials — that rule stays intact. Extract is an offline producer, and TAGURU_EXTRACT_* lives only in its own process environment. It is exactly a custom agent-side pipeline holding its own keys, just shipped in the box as a subcommand. It never touches the data directory (and takes no lock). Its only output is files.

There is one wire protocol, deliberately the same stance as embeddings: OpenAI-compatible. https://api.openai.com/v1/chat/completions works as-is, and so do local servers (Ollama, llama.cpp, vLLM). Bedrock and native Anthropic are bridged by any proxy that speaks /chat/completions, such as LiteLLM — the same bridge pattern the Bedrock guide shows for embeddings. taguru communities reads the same TAGURU_EXTRACT_* variables for its community summaries — one provider configuration for every LLM the toolchain talks to.

Local models: five field notes

All of the following was learned by actually running the pipeline against Ollama on a laptop.

• Turn thinking mode OFF. A model in reasoning mode spends the whole time budget on invisible thinking tokens before the first byte of JSON: the same 10 KB document that extracted in ~30 s with thinking off blew through a 300 s timeout with it on. The extractor speaks plain OpenAI-compatible chat and does not toggle vendor-specific thinking flags — pick a non-thinking model, or disable thinking on the serving side. The symptom: a busy GPU and timed out reading response — or, with no timeout set, an empty response after minutes of thinking (reported as exactly that). Thinking exhausted the generation budget.
• Give the server a real context window. A 24 KiB chunk is roughly 6k tokens. A serving default of 4k truncates the request silently — no error, just quietly degraded output. On Ollama, bake the window into a derived model: FROM <base> + PARAMETER num_ctx 16384 → ollama create.
• Match the timeout to the hardware. TAGURU_EXTRACT_TIMEOUT_SECS bounds each chat completion (default 300; 0 = unlimited). Every attempt gets the full budget, and a transient failure is retried up to 4 attempts total, so a provider that stalls instead of erroring can cost several multiples of the timeout per chunk before the retries exhaust — worth remembering before dropping the timeout to something tight. Under the escalation ladder (--max-output-tokens or --structured-output) a timeout is not retried at the same size at all: the chunk splits instead (ADR 0020), so the cost is one timeout per split level, and --chunk-bytes lets you start smaller.
• The manifest trusts the model's name. Re-point a serving alias at a different base (overwrite ollama create my-extractor with another base) and nothing the manifest can see changes — documents stay "unchanged" under what is actually a new model. Name models honestly, or --force after re-pointing. The same caveat as the embedding vector cache (also keyed by model name).
• A wall of a malformed answer near the output cap can stall a chunk for minutes. A small local model that runs its output budget out mid-answer produces one huge invalid response; replaying that whole response back verbatim and then asking for the same length again just reproduces the same cutoff — one chunk of a few hundred bytes measured at 4-5 minutes this way. The first-class fix is --max-output-tokens: with an explicit budget, a cut-off answer takes the deterministic escalation ladder (below) instead of a shorter-please retry. Without one, --fact-budget N asks for fewer associations per chunk up front, the truncation-aware correction (see below) asks for a shorter answer once the provider's finish_reason says the prior one was cut off, and TAGURU_EXTRACT_CORRECTIVE_CONTEXT_BYTES=0 is the blunter fallback: never replay the bad answer at all.

These five are the extraction side of the same serving layer. The same knobs — thinking mode, context window, output caps, and streaming — seen from the answering side, plus where to separate Taguru's own latency from the model's, are in Troubleshooting — the serving layer. All of it applied together, against Ollama, extract through answer, is the local RAG walkthrough.

What the model is asked, and what is enforced on this side

The system prompt is the /protocol ingest loop distilled for a producer that cannot resolve live: one fact per association, short names in the document's language, one spelling per referent, negation as negative weight, no paraphrase re-assertion, membership edges made explicit, procedures chained with a single next-step label. The document is presented with every paragraph numbered (blank-line boundaries counted across the whole document, not per chunk — the same numbering --questions uses), and each association is asked to tag its paragraph of origin. That tag rides into memory with the fact; nothing downstream persists it yet. Relation labels converge across the run: each document's labels are offered to every later document's prompt (the offline stand-in for check-before-mint), so one run doesn't mint synonyms per file but converges on one vocabulary — ranked by how many associations (plus any alias that settled on it as canonical) have used each one so far, most-reused first, with a (×N) count on anything used more than once (issue #759): a label many associations already share is a safer reuse than one that showed up once, and the count is the signal that tells them apart. Temperature is 0.

When --schema names a document whose mode is not "off", the prompt gains one more block after the vocabulary one: the allowed entity type names, one label: domain → range line per constrained relation (budget-capped like the vocabulary block, live-vocabulary relations first), and an instruction to assert types on the reserved schema:type label — see the context schema reference for what that label means. It is deliberately never rendered as a JSON Schema enum, so a schema-constrained model can still propose a genuinely new relation rather than being forced to misuse an existing one. A domain/range violation or a closed_labels refusal in the answer earns its own targeted corrective turn (schema_output_issues, a sibling of the contract violations below), separate from — but bounded by the same TAGURU_EXTRACT_MAX_ATTEMPTS as — the rest of this section. Adding this block bumps PROMPT_VERSION 2 → 3; see the manifest for what that invalidates.

Model output is treated as untrusted input, and the contract is enforced on this side of the wire:

• Exact duplicate triples fold into one line (the in-document paraphrase rule, applied mechanically).
• Items that could never import as answered are removed mechanically, before any corrective turn (ADR 0013): an association or alias with a required field missing or empty, an alias that maps a spelling to itself, an alias whose canonical names nothing the document's associations contain, an alias whose spelling an earlier document of the same run (or the --vocabulary context) already settled on as a different concept or label — import would refuse that rewire and stop the stream, so it never gets written — a relation label that is a single character (most often a bare particle picked up as the whole relation; two or more characters survives — same anchor-nothing judgment as a single-character candidate name), and a subject/object that never appears in the document text (a whitespace- and case-blind check, so a normalized compound like プール最大接続数 for a document that says プールの最大接続数 still passes). Every removal is named path-first on stderr, counted on the report line as removed (mechanical validation), and listed in the diagnostics sidecar — never a silent drop. One removal class is the exception to "before any corrective turn": an alias whose in-document shadowing or conflict the one Stage 2 corrective turn leaves standing is removed after that turn (ADR 0022), with the same accounting.
• Invalid items removal cannot judge — wrong-typed or oversized fields, zero/non-finite/over-cap weights, unknown alias kinds, shadowing or conflicting aliases, out-of-range questions — earn a path-addressed corrective turn by default (see below): a present-but-wrong value is content the model can actually fix.
• A missing or out-of-range paragraph tag loses only the tag, never the fact — the association keeps the model's subject/label/object/weight judgment as-is.
• An alias is adopted only when its canonical is a name this file's associations register, and never when the alias spelling is itself such a name (either would fail the batch at apply time).
• Every file written is re-parsed with the import parser before it is written — extract cannot produce a file import would reject.

--candidates (ADR 0014, #496 S2) attacks spelling variance one step earlier, before the model answers at all: the document's own names — kanji/katakana compounds split at hiragana boundaries, script-adjacent mixtures like 約40分, ASCII identifiers like cargo-nextest — are segmented deterministically (no dictionary, no model, the same list on every run) and offered in the system prompt as preferred subject/object spellings. The block is non-restrictive by contract: the model is asked to reuse a listed spelling when one names the entity it means, and explicitly told that entities outside the list stay allowed — constraining spelling never becomes constraining what may be extracted. Off by default; toggling it (a computation input) re-extracts. All-hiragana nouns and multi-word Latin noun phrases are known blind spots of the dictionary-free segmenter — ADR 0014 documents the limits and the measured path to a morphological analyzer if they ever cost recall. Pair it with --structured-output: under a candidate block, a mid-size model was measured emitting the offered spellings as unquoted JSON values — a syntax failure class constrained decoding removes structurally, which is exactly the syntax/naming role split the two controls are designed around.

--vocabulary (ADR 0015, #496 S3) is the cross-document half of the same idea: point it at an exported batch stream of the target context (taguru export --out DIR, or GET /contexts/{name}/export saved to disk — a file or a directory of files) and the context's own concept names and relation labels are offered as preferred spellings, with one extra instruction: use the context's exact spelling even when the document spells the same entity differently. That is the cargo-nextest/nextest twin being prevented at answer time instead of detected by the consolidation audit after import. Alias spellings are never offered (they are the variants a canonical exists to fold), harvested labels seed the "relation labels already in use" block from the first document, and a context spelling never fails the mechanical occurrence check — the steering and the validation agree by construction. The harvested name set is content-digested into the manifest, so refreshing the export re-extracts exactly when the names actually changed. Prompt-side the concept list caps at 200 names (alphabetical, deterministic); relevance-ranked selection is ADR 0015 §4's measured upgrade path. Re-export the context periodically — a stale vocabulary still steers correctly toward every name it knows and simply misses newer ones.

--coverage (ADR 0016, #496 S4) is the recall-side mirror of the mechanical validation above: where the occurrence check removes what the model asserted and the document never said, coverage reports what the document said and the model never asserted. Every sentence holding two or more of the document's own names (the --candidates segmentation, applied whether or not that flag is on) is owed a triple; when no extracted association lands at least two of its three parts (subject/label/object) in the sentence, the sentence is flagged — one stderr line each (uncovered: [paragraph N] <sentence>), a count on the report line, an uncovered count in the diagnostics document record. Report-only: the batch is written whole either way, no extra model call is made, and the flag is not a computation input — a rerun over an unchanged corpus skips every document and still reports, judging each from the batch it already wrote, so any past run's recall ceiling is measurable for free. A gap is a lead, not a verdict (a name-dense heading can flag without a real miss); re-extracting flagged sentences automatically is ADR 0016 §4's measured upgrade path.

A model that answers with anything but the JSON object gets a corrective turn: its own bad answer goes back to it, followed by a request to try again. By default that happens once — 2 total attempts per chunk, today's fixed behavior — and TAGURU_EXTRACT_MAX_ATTEMPTS raises the ceiling to anywhere from 1 (no corrective turn at all) to 10. Every corrective turn rebuilds the conversation from the system/user base plus only the single most recent bad answer, never the accumulated history, so TAGURU_EXTRACT_CORRECTIVE_CONTEXT_BYTES bounds the replay the same way on attempt 5 as on attempt 2: left unset the bad answer is replayed back in full (today's behavior), a byte count truncates the replay to that size, and 0 omits it behind a placeholder entirely. When the provider's own finish_reason says the bad answer was cut off at its output-length cap, the corrective ask itself changes: instead of repeating the request that just produced a too-long answer, it asks for a shorter one, naming --fact-budget when the run has one — repeating the identical ask just reproduces the identical cutoff, which is the stall field-noted above. That shorter-please swap is the no-budget-control fallback: a salvaged-shorter answer measurably costs most of the extraction, so once --max-output-tokens or --structured-output engages the ladder below, a cut-off answer is regenerated or split instead — never asked to shrink.

A syntactically valid answer that still fails the contract above first goes through the mechanical pass (ADR 0013): removable items come out with their removals recorded, and only if issues remain that removal cannot judge does the answer earn a corrective turn — the corrective turn is the last resort, not the first response. That demotion is measured, not aesthetic: on the failure corpus that motivated it (empty object fields, self-referential aliases), corrective turns burned 53–63 seconds per document across five attempts without fixing anything, while removal is instant, deterministic, and loses nothing a corrected answer would have kept. The corrective treatment itself is bounded by the same TAGURU_EXTRACT_MAX_ATTEMPTS/ TAGURU_EXTRACT_CORRECTIVE_CONTEXT_BYTES controls — never a second, unbounded mechanism. The turn names every violation by its exact JSON path (for example associations[1].weight: expected finite non-zero number, got string "strong") and asks for the complete corrected object: keep every item, correct the named fields rather than deleting their items, add nothing that wasn't already there, JSON only. A shadowing or conflicting alias can only be judged once every chunk of the document has answered — that check runs once per document, right before the facts are merged, and (like the per-chunk check) spends at most one corrective turn; an alias issue that turn leaves standing is then removed with accounting rather than failing the source (ADR 0022: an alias records a spelling variant, never a fact, so losing one costs nothing the consolidation audit cannot propose later — while failing the document cost every fact it held; a standing issue about an association, such as a schema domain/range violation, still fails the source); a dangling alias found at that same point is mechanically pruned instead, after any corrective turns, since an alias whose canonical resolves to nothing cannot import at all — and so is an alias that would rewire a spelling an earlier document of the run already claimed, which the model cannot correct (it can re-judge its own associations, never un-claim a previous document's name). If the corrected answer is still invalid, or the model answers with something cut off, refused, or empty, the source fails outright and nothing is written — a batch is only ever the source's complete, valid truth, never a subset quietly missing the items that didn't fit. --lossy (or TAGURU_EXTRACT_LOSSY) opts back into the pre-this-behavior default: invalid items are dropped and counted without ever costing a corrective turn, and the one-line report marks every such run's drops with (--lossy) so a policy trim (a --questions cap overflow, a volunteered question nobody asked for) is never confused with a discarded fact.

This corrective-turn policy is a separate layer from transport retries: a transient provider failure (429, 5xx, transport) is retried up to 4 attempts total regardless of TAGURU_EXTRACT_MAX_ATTEMPTS, waiting a full-jitter exponential backoff between them — 1 s doubling toward a 30 s ceiling, randomized so concurrent chunks under --parallel don't all wake up and retry in lockstep. A 429 that carries Retry-After as delta-seconds uses that delay instead, verbatim (clamped to the same 30 s ceiling), since the server's own instruction beats a guess; HTTP-date values are not recognized and fall back to the computed backoff. A non-retryable 4xx fails immediately, spending none of the retry budget. Beyond the last attempt, of either kind, the document fails, the remaining documents continue, and the run exits 1 — a re-run re-extracts only the failures, because the manifest records only successes.

Structured output and the output budget

Not to be confused with --schema above. Everything in this section is about response_format on the OpenAI-compatible wire — how the model's answer is shaped. --schema is Taguru's own context schema document — what types and relations are allowed to mean in the target context. The two names collide; the concepts do not.

By default nothing above changes: the request stays the plain {model, temperature, messages} it has always been. Two opt-in controls put the reliability strategy of ADR 0001 on the wire; engaging either switches the length handling from the shorter-please correction above to the deterministic ladder below.

--structured-output selects a rung of the capability ladder. json-schema sends the canonical extraction schema as response_format: {"type": "json_schema", …, "strict": true} — a backend with constrained decoding (Ollama, llama.cpp, vLLM) then cannot answer with anything but the schema's shape. json-object sends JSON mode: syntax forced, shape not. auto probes the endpoint once at startup — a tiny completion carrying exactly the extraction response_format, with an ask that invites prose — and keeps the strongest rung the answer verifies, falling from json_schema to json_object to bare prompted JSON and reporting the resolution on stderr. A backend may accept a parameter without honoring it, which is why auto verifies behavior rather than trusting a 200. The pinned modes skip the probe and trust the operator: a backend that rejects the parameter fails loudly on the first document. One caveat by design: OpenAI's own strict mode requires every property listed in required, which the canonical schema's optional weight/paragraph deliberately are not — against api.openai.com, auto therefore lands on json_object today (a strict-mode schema variant is a possible future refinement). A schema-constrained answer still passes through every check in the contract above — the wire narrows shapes; validation stays the authority — and one that fails anyway is reported on stderr as provider non-conformance. A probe that passes says nothing about a real document: some local models loop under constrained decoding — the tiny probe answers cleanly, then a 2 KB abstract generates 15,000 tokens of schema-shaped output and never stops (measured on Ollama with a 30B MoE). So auto also demotes at run time (ADR 0021): when a chunk exhausts the ladder under a constrained rung — length at the budget and again at the escalated resend, or a timeout — the run drops one rung (json_schema → json_object → prompted JSON), reports it on stderr, and restarts that chunk at the ladder's top; only a chunk that exhausts the ladder with nothing left to demote splits. The demotion is run-wide and never reverses: the finding is about the backend, not the chunk. The pinned modes never demote — a looping model under json-schema splits down to the floor and fails, so pin json-object or off for a model known to loop.

--max-output-tokens N makes the output cap an explicit request parameter instead of ambient serving configuration. When a completion still ends with finish_reason: length, the next action is deterministic, and it is never "re-ask under the very limit that just proved too small": first the budget escalates once — the same ask resent neutrally at TAGURU_EXTRACT_ESCALATION_FACTOR times the budget (default 2×; 0 sends no cap at all), the truncated answer discarded (its prefix is never salvaged, even when it happens to parse); if the escalated ask still overruns, the chunk splits — each half re-labeled with its paragraph numbers and run through the ladder from the top at the configured budget; a piece too small to split fails the source with a named diagnosis rather than importing a truncated extraction. The escalated resend is capped on purpose (ADR 0019): a model that loops under constrained decoding never ends an uncapped resend with length — it ends with the TAGURU_EXTRACT_TIMEOUT_SECS timeout, retried as a transport failure, and the split rung is never reached (measured at 10–25 minutes per chunk on a looping local model). Capped, the loop ends with length within a couple of the budget's wall-clock and falls through to the split. Raise the factor for a provider whose honest answers routinely need more than twice the budget; set it to 0 only for one that is known not to loop. A completion that runs TAGURU_EXTRACT_TIMEOUT_SECS out under the ladder is treated the same way as length (ADR 0020): the piece is too big for the time budget, so it goes straight to the split rung — never retried at the same size (the split is the retry; transport failures, 429 and 5xx keep their four attempts), never escalated (a larger output cap cannot make a slow piece faster) — and at the split floor it fails the source with the timeout named and the two knobs that would have helped. Policy refusals (finish_reason: content_filter) are terminal immediately — no corrective turn argues with a policy. Both controls are manifest compute inputs like --questions: changing them re-extracts.

Chunking

Documents are sent in chunks of at most --chunk-bytes (24 KiB by default; TAGURU_EXTRACT_CHUNK_BYTES sets the default), cut at paragraph boundaries (aliases and duplicate folding stay coherent across chunks — an alias in chunk 1 whose canonical first appears in chunk 3 still lands). A fact whose evidence straddles a chunk boundary can be missed; that is why the cap leans large. Passages are never chunked: the batch carries the full document text as-is. Documents over 8 MiB are refused — the batch passage could not carry them. Split the document. Under the escalation ladder (above) a length-limited or timed-out chunk can split further at run time, so --dry-run's chunk count is a lower bound on model calls, not an exact one. The cap is a computation input: a document extracted under a different --chunk-bytes re-extracts (the default cap matches manifests written before the flag existed). Lower it for a slow local provider or for output-dense documents — statutes, minutes — whose answers outrun the time or token budget at the default cap (ADR 0020).

--parallel N (or TAGURU_EXTRACT_PARALLEL; the flag wins when both are set) runs up to N of one document's chunk completions concurrently instead of one at a time — the chunks still merge in their original order, so output is identical to a sequential run, just faster wall-clock. The first chunk to fail (by index, not by which thread finishes first) still fails the document: no worker claims a new chunk once the failure is recorded, but a chunk already claimed and in flight at that moment still runs to completion — its result is simply discarded. Parallelism never crosses documents — each document's relation-label vocabulary is offered to the next document's prompt, so running documents concurrently would race that hand-off and let labels diverge between them. Fanning out chunks multiplies request pressure on the provider, which is also why the retry policy above leans toward more attempts and jittered spacing rather than fixed timing.

Questions (doc2query)

With --questions N, the same extraction call also proposes up to N realistic retrieval questions per paragraph — questions the paragraph answers, phrased the way a user would type them, kept away from the paragraph's own wording — and the batch carries them as question lines. Questions ride the same server-numbered paragraphs as the associations, so the indexes written out match exactly what the server validates and cannot drift. Generation riding the extraction call is deliberate: there is no cheap questions-only pass, so changing N means re-extracting (the manifest records it as a compute input). Every server indexes each question's terms into its paragraph's BM25 postings, so question-shaped searches land lexically; a server with TAGURU_EMBED_PASSAGES additionally embeds each question next to its paragraph.

The manifest: skipping what's already extracted

Extraction being the expensive stage, --out carries an .extract-manifest.json: per document, the "content hash × model × prompt version × target context" its batch file was computed from. Documents whose compute inputs are unchanged are skipped, --force overrides, and a missing or corrupt manifest degrades to re-extraction — never to a false "unchanged". Keep the out directory between runs and a nightly extract-and-import pays model calls only for changed documents; import's retract-then-apply makes the re-application exact.

PROMPT_VERSION is one of the compute inputs the hash covers, so it bumping (2 → 3, for the schema vocabulary block above) invalidates every manifest entry computed under the old prompt — the next run against an existing --out re-extracts everything once, whether or not --schema is even in use.

The context being a compute input is deliberate: it is baked into every header written, so re-running the same out directory with a different --context re-extracts instead of leaving files aimed at the old destination. Still, use one out directory per context — the manifest has one entry per document, so alternating two contexts in one directory re-extracts everything on every switch and the later context's files overwrite the earlier's. Batches from several out directories can be applied together in one import (taguru import sake/ code/) — every file's header names its own context, so "extract once per context, import everything at once" is the intended shape.

Chunk checkpoints: surviving an interruption mid-document

The manifest above protects a whole document that hasn't changed — but a document is not the smallest unit of work. A long document can take many chunk completions, and until the last one lands, an interruption (Ctrl+C, a preemptible instance reclaimed, a later document's panic) threw away every chunk already extracted for it. --out also carries a .extract-checkpoints/ directory, one JSON file per document, holding every chunk unit successfully extracted so far. A rerun over the same --out picks up mid-document: chunks already checkpointed are never re-sent to the model.

Each checkpointed unit is keyed by the hash of its own text, not by chunk index. This matters because the ADR 0001 §7 length ladder can split an oversized chunk into smaller sub-pieces on its own recursive re-attempt — a resumed run's split points are not guaranteed to line up with a prior run's. Keying on content hash instead of position means a completed sub-piece is recognized as done regardless of how either run divided the chunk around it, and a not-yet-completed sub-piece is correctly treated as new work. That same hash is the unit's piece_id in the trace, and since 0.9.5 each unit also records the completion that produced it (attempt: {run_id, attempt_seq}), so a resumed document's trace still names the original run for every reused unit.

A checkpoint file also carries the same compute-input fingerprint the manifest checks (content hash, model, prompt version, context, --questions, --fact-budget, --structured-output, and so on). Any mismatch — the source edited, a setting changed — invalidates the whole file, the same "unreadable or incompatible degrades to re-extraction, never to a false reuse" posture the manifest takes. --force discards a document's existing checkpoints outright, one level deeper than bypassing the manifest skip. A document whose batch lands successfully has its checkpoint file deleted; a document that ultimately fails (Stage 2 correction, merge, self-validation) keeps it — the chunks it did complete are still good work worth keeping for the next attempt, and the failure line says how many units are checkpointed and that a rerun without --force resumes from them, re-asking only what failed. There is deliberately no "write what succeeded" mode: a batch is the source's complete, valid truth or nothing (ADR 0001 §8), and the checkpoints already make the retry cost only the failed part.

--dry-run reads checkpoints too (never writes or calls anything) and folds a "reusable" count into its existing per-document line, e.g. "3 chunk(s), 1 reusable from checkpoint". The count is top-level-chunk granularity only — dry-run resolves no ladder, so a chunk that would end up split on a real run is honestly reported as pending rather than guessed at.

Ctrl+C (or a SIGTERM) during a run is cooperative: the first signal finishes the chunk in flight, checkpoints it, and stops before starting the next one — between chunks within a document under the default sequential mode, and between documents under --parallel (its concurrent chunk dispatch is not interrupted mid-flight, so stop takes effect at the next document boundary rather than the next chunk). The process exits with code 130 and prints a line naming how many documents were reached before stopping; rerunning the same command resumes from the checkpoints just as after any other interruption. A second signal forces an immediate exit, also with code 130, for when the graceful path is itself stuck (a hung connection, a provider that never returns).

Operationally, this is what makes a long extraction job viable on a spot/preemptible instance: schedule it to run against a persistent --out, let the scheduler kill it whenever the instance is reclaimed, and requeue the same command — the manifest skips finished documents, checkpoints skip finished chunks within the document that was in flight, and nothing already paid for is ever paid for twice. Composing this with taguru import's own atomicity into a full run/interrupt/resume operating picture — bounded windows, work-unit enumeration, failure/re-submission — is the long-running ingestion guide.

Diagnostics sidecar

By default a failed document earns one stderr line and nothing else — enough to know that a document failed, not enough to tell a truncated answer apart from a model syntax error or an empty-answer thinking burn. --diagnostics-out FILE (or TAGURU_EXTRACT_DIAGNOSTICS; the flag wins when both are set) opts into a JSONL sidecar, written incrementally — a killed run keeps every record already flushed. It is off by default, and without it stdout/stderr are unchanged byte for byte: the sidecar is purely additive.

The sidecar is a tagged stream: a kind field discriminates "run" (the first line — run_id, the identity of this invocation; ADR 0023), "chunk", "attempt", and "document" records, described below. A consumer written before "chunk"/ "document"/"run" existed keeps working unmodified as long as it filters on kind == "attempt" — the same discriminator it already needed to read the sidecar at all — since an attempt record's own shape has only ever gained fields.

Every attempt record carries: run_id, attempt_seq (one-based, in issue order over every extraction completion of the run — the corrective budget's own attempt below restarts per round, this never does) and piece_id (the sha256 of the text this completion asked about — the chunk, or the split sub-piece; for cross_chunk, the piece whose answer is being corrected), the three keys the trace file joins on; source, chunk_index, attempt (one-based), max_attempts, stage (item for the per-chunk loop, cross_chunk for the Stage 2 alias correction above), state — the ADR 0001 §7 vocabulary this producer classifies every attempt into before any parsing happens: stop_valid, stop_malformed, length_limited, empty, refusal, timeout, transport — length_limited, parse_error (the diagnosis, present for every state but stop_valid), validation_issues (path-addressed, when the answer parsed but failed the contract — under --schema this includes "domain"/"range" entries from schema_output_issues, the same issue vocabulary the server's own write-time check uses), removed_items (path-addressed, on a stop_valid attempt whose accepted answer had items mechanically removed — ADR 0013; omitted when nothing was), piece_bytes and requested_max_tokens (escalation ladder only, omitted otherwise: the byte length of the split sub-piece this round asked about — sub-pieces share one chunk_index — and the max_tokens the round sent, when one was), elapsed_seconds, and a nested provider_metadata (finish_reason as received, input_tokens/output_tokens/total_tokens when the backend reports usage — null for timeout/transport, where no response ever arrived). Four transport-layer retries inside one attempt (see above) are one record, not four: the sidecar reports at the extraction level, matching what a chunk actually experienced. Field names mirror the Python LangChain SDK's event stream (ProviderMetadata, AttemptFailed) wherever the concept matches, so tooling built against one producer's diagnostics reads the other's.

One chunk record is written per chunk, before that chunk's first attempt: source, chunk_index, chunk_total, chunk_sha256 (the chunk text as sent to the model), chunk_bytes, and paragraph_first/paragraph_last (inclusive) — the range of the server's own canonical paragraph numbering (the same locator a batch's paragraph field, passage store, BM25 lane, and vector lane all already share) that chunk covers. This is a paragraph-index range, never a byte offset: chunking runs on a relabeled rendering of the document, not the original bytes, so a byte offset into it would not resolve to anything a reader has on disk. Given (source, paragraph) and the document's own bytes, a byte range and the verbatim text follow from one deterministic, offline call to the server's own paragraph splitter — see ADR 0003 §7. An oversized paragraph that itself straddles more than one chunk repeats its number across each one, rather than guessing at a boundary that was never byte-addressable to begin with.

One document record is written per document, once it lands successfully — a structured counterpart to the single human-readable summary line every run already prints: source, batch_path, associations, concepts, labels (counted separately, unlike the summary line's combined "alias(es)" figure), questions, duplicates, dropped, removed (the mechanical-validation count the summary line reports — ADR 0013), and uncovered (the coverage-gap count under --coverage — ADR 0016; 0 when the flag is off). A document that fails never reaches this record — its absence marks exactly that, the same "absence marks incomplete" convention --diagnostics-out consumers should apply to any record kind this sidecar might add in the future.

Raw model text is never captured by default — metadata only. Chain-of-thought is never captured at all, under any setting. TAGURU_EXTRACT_DIAGNOSTICS_RAW_BYTES opts into the model's final answer text on each attempt record (the only kind that carries one at all) — the full text of every prompt and answer lives in the attempts log beside the batch instead — byte-capped at capture (the same treatment TAGURU_EXTRACT_CORRECTIVE_CONTEXT_BYTES gives a replayed bad answer above) — because a raw response can embed the source document's own content, an uncapped default would leak document text into a file meant for troubleshooting model behavior.

The sidecar is truncated fresh on every run — it describes this run, never a log appended across runs — so a rerun the manifest skips entirely leaves it empty, and a path inside --out risks collision with a batch file or the manifest itself. Keep it outside --out. It is a diagnostic aid, never a source of truth: the manifest stays exactly what it has always been, a skip-index of successes, with no diagnostics mixed in. For watching this sidecar (or the SDK's equivalent event stream) live against a multi-hour run, see long-running ingestion.

Trace: from any batch item back to the text and the completion that produced it

Every written batch gets a sibling under <out>/.extract-trace/, named like the batch file, always on, no flag (ADR 0023). Where the diagnostics sidecar describes a run, the trace describes a batch: it is written atomically in the same step as the batch, replaced when the document is re-extracted, left alone when the manifest skips it, never written for a document that fails (its checkpoint keeps what matters for the resume), and never a manifest input — a batch from before this file existed simply has none. It is hidden and in a subdirectory so taguru import DIR never reads it as a batch. The batch format itself is unchanged: import refuses unknown fields, so an item is identified by its content, exactly as the batch makes it unique — the association triple, the alias spelling within concept/label, the question's (paragraph, text).

The file is a tagged JSONL stream like the sidecar (kind; skip kinds you do not know), in this order: one document (run_id, source, document_sha256 — the manifest's hash of the text, batch_path, chunk_total); one chunk per chunk of the plan (the same chunk_index/chunk_sha256/ chunk_bytes/paragraph_first/paragraph_last the sidecar's chunk record carries); one piece per answer the chunk loop kept — a chunk, or each sub-piece the split rung made of one, so several pieces can share a chunk_index — with piece_id (sha256 of the piece text as sent; the checkpoint unit's key, so an unsplit chunk's piece_id equals its chunk_sha256), chunk_sha256, piece_bytes, its own paragraph range, reused (the answer came from a checkpoint rather than a completion of this run), and attempt — {run_id, attempt_seq} of the completion whose answer this piece's output is: the accepted Stage 1 answer, or the Stage 2 corrective answer when one replaced it; for a reused unit, the completion of the run that produced it, which the checkpoint now records (null only for a checkpoint written before 0.9.5); then one item per batch line in batch order (item: association | concept | label | question, the content key, and piece_id); then one steering record (after document — listed here in reading order) with what taguru itself put into the prompt, as data (ADR 0027): candidates (ADR 0014's list as offered), vocabulary ([{label, count}] in prompt order — the reuse list's own ranking and cap, computed by the same code that renders the block), context_names (--vocabulary's capped list), and schema ({types, constrained_relations}; null when no schema block was prompted); chunk_index is null — document scope, every chunk sees the same lists today; per-chunk context (a future control) adds records with the index set. Then one loss per item the model's accepted answer held that the batch does not (ADR 0024): item (association | alias | question), reason — removed (mechanical validation, ADR 0013, including the Stage 2 alias prunes), dropped (the contract refused it as written: an empty name, a zero weight, a dangling alias, a question over the --questions cap… and, under --lossy, an array element that was not an object, dropped at parse), or duplicate (an identical item was already kept) — rule in the report's own words, path (removals only: the item's address in the answer), raw (the item exactly as the model wrote it), piece_id, attempt, paragraph, and text: the original text the item was about — the cited paragraph when the item cited a valid one, else the whole piece the model was shown. A duplicate adds kept_piece_id. Removals come first, piece by piece, then merge's drops and duplicates; the document record's removed/dropped/ duplicates counts are these records' lengths by reason. The loss rate — lost ÷ (kept + lost) per piece, rolled up — follows from the item and loss records alone. After the losses come the document's own side (ADR 0026): one paragraph record per canonical paragraph — paragraph, bytes, items (kept associations and questions citing it), covered, and text present exactly when nothing cites it, so the unreflected paragraphs are listable in the original and the paragraph coverage rate (count- or byte-weighted) is one fold over the records — and, under --coverage, one uncovered record per coverage gap, line for line with stderr but carrying the full sentence (stderr's quote stays byte-capped), the paragraph's text, and the owning chunk (chunk_index/chunk_sha256).

taguru anchoring OUT_DIR --json report.json judges the written batches against their own passage text — the anchoring rate (strict and alias-group) and locator validity, hallucination's mechanical floor; batch files are all it needs, so pre-0.9.5 output qualifies (taguru anchoring --help). scripts/extract_metrics.py (python3, standard library only) folds these records — and the attempts log below — into the #784 metric tables per document, context, group, and run (loss rates by reason, paragraph coverage, correction success, attempt-state and ladder-move counts, label concentration, graph shape, and time/token/money cost, with --compare for run-to-run deltas); --help documents the ledger and price inputs. Joins: item → piece by piece_id; piece → chunk by chunk_index or chunk_sha256; piece → the sidecar's attempt record by (run_id, attempt_seq) when that run's sidecar was kept; any record → the document by the file itself. A duplicate triple two chunks both answered is attributed to the copy merge kept, the first — the batch holds one line, so the trace holds one item. Given a piece's paragraph range and the document's own bytes, the text the model was shown follows from the server's paragraph splitter exactly as for the sidecar's chunk record (ADR 0003 §7). A trace that cannot be written earns one stderr line and never fails the document: the batch and the manifest are the truth, the trace is advisory.

The attempts log: every prompt and every answer, in full

Beside the trace, <out>/.extract-trace/<batch stem>.attempts.jsonl keeps the text of every conversation the document had with the model, by default (ADR 0025): one document record per run over the document (run_id, source, document_sha256, resumed); the system prompt once, in full, as a system record (sha256, bytes, content) — it is fixed for the document, so attempts name it by hash; and one attempt per completion, in issue order, carrying the sidecar's own fields (run_id, attempt_seq, piece_id, stage, attempt, state, finish_reason, token counts, …) plus messages — every turn as sent, the system turn as {role, system_sha256} and the rest as {role, content} in full, the replayed prior answer and corrective ask of a retry included — and answer, the model's final text in full (null for timeout/transport). Every attempt record also carries transport_retries — how many transport-layer tries (429, 5xx, transport errors) failed inside this one attempt before its outcome (0 = a clean first try; the four-retries-are-one-record ruling is unchanged, the count makes it visible). A corrective attempt also carries corrects — the {run_id, attempt_seq} of the attempt whose answer it replays and asks to fix (Stage 1: the same piece's previous attempt; Stage 2: the accepted attempt whose output is being corrected; absent exactly when that attempt has no recorded identity — a Stage 2 correction of a unit reused from a pre-0.9.5 checkpoint, written before attempts had ids) — so the correction tuple "what was flagged → what was asked → what came back → what was adopted" is a join over records already on disk (ADR 0028): the flagged issues sit on the corrected attempt, the ask is the corrective attempt's own last turn, adoption is the trace piece record naming this attempt, and what was flagged-then-removed instead lands in the trace's loss records. Nothing is capped. The ladder's own actions land beside the attempts as kind: "move" records (ADR 0029): move is escalate (ADR 0019; from_max_tokens/to_max_tokens), demote (ADR 0021; from_rung/to_rung), or split (ADR 0001 §7 / ADR 0020; piece_bytes/split_cap/sub_pieces, the reason telling the output cap from a timeout), each with run_id, piece_id, chunk_index, and reason in the stderr line's own words — so per-document-kind frequency and cost of every retry-machinery action is a fold over this file. It is written incrementally (a killed run keeps every completion made), kept when the document fails — that is what it is for — and, on a checkpoint resume, appended to rather than truncated, so it spans exactly the runs that built the batch; a fresh start (--force, a changed document) truncates it. Joins: the sidecar by (run_id, attempt_seq), the trace's piece by piece_id and attempt, a loss's path into answer. Measured on the 0.9.3 field corpus it is 5–70 KiB per document. TAGURU_EXTRACT_TRACE_ATTEMPTS=off switches it off. The diagnostics sidecar is unchanged: metadata only, TAGURU_EXTRACT_DIAGNOSTICS_RAW_BYTES still the capped opt-in — the sidecar is the file to hand to someone else; the attempts log stays with the data, exposing no more of the document than the batch and checkpoint beside it already do.

Reading the records: tracing one item, and what to adjust

Everything above says what is written. This section is the reader's path through it (#784): start from one line of a finished batch and end with the original text, the exact prompt, and the model's full answer side by side — then, when a number looks wrong, which knob answers it. jq is enough; no taguru command is involved. Throughout, for a batch out/a.jsonl:

t=out/.extract-trace/a.jsonl               # the trace
l=out/.extract-trace/a.attempts.jsonl      # the attempts log

From one batch line back to everything that produced it

1. The item. A batch line is identified by its content, so select the trace item record with the same key — the association triple, the alias spelling, or the question's (paragraph, text):

jq -c 'select(.kind=="item" and .subject=="酵母"
  and .label=="作る" and .object=="アルコール")' "$t"
# → {"kind":"item","item":"association",…,"piece_id":"3f2a…"}

(An alias selects on (.item=="concept" or .item=="label") and .alias==… and .canonical==… — a concept alias and a label alias are separate namespaces, each its own item value; a question on .question and .paragraph.)

2. The piece. piece_id names the text the item came from:

p=3f2a…   # from step 1
jq -c --arg p "$p" 'select(.kind=="piece" and .piece_id==$p)' "$t"
# → chunk_index, piece_bytes, paragraph_first/last, reused,
#   attempt: {run_id, attempt_seq}

attempt is the completion whose answer this piece's output is; paragraph_first/paragraph_last locate the text in the original document by the server's canonical paragraph numbering.

3. The conversation, in full. The attempts log has every completion that asked about this piece, in issue order — the accepted one and every failed or corrected try before it. The accepted one is the record matching step 2's attempt on both run_id and attempt_seq: a resumed document's log spans several runs, and attempt_seq restarts per run, so alone it is ambiguous:

jq -c --arg p "$p" 'select(.kind=="attempt" and .piece_id==$p)
  | {run_id, attempt_seq, state, messages, answer}' "$l"
jq -r 'select(.kind=="system") | .content' "$l"   # the system turn, by hash

The user turn inside messages is the exact text the model was shown — the original passage needs no separate lookup — and answer is the reply in full. state, finish_reason, token counts, and transport_retries ride on the same record.

4. The correction chain. A record carrying corrects is a corrective attempt; its target is the record with that {run_id, attempt_seq}. The whole tuple — what was flagged, what was asked, what came back, what was adopted — reads off records already in hand: the flagged issues are the corrected attempt's validation_issues, the ask is the corrective attempt's last messages turn, the reply is its answer, and adoption is the piece record from step 2 naming (or not naming) it:

jq -c 'select(.kind=="attempt")
  | {run_id, attempt_seq, stage, state, corrects,
     issues: .validation_issues}' "$l"

5. The losses. What the accepted answer held that the batch does not, each with the item exactly as the model wrote it (raw) beside the original text it was about (text) — read the pair before concluding anything about why:

jq -c --arg p "$p" 'select(.kind=="loss" and .piece_id==$p)
  | {reason, rule, raw, paragraph, text}' "$t"

6. The document's side. The reverse direction — what the output does not reflect — is the paragraph records (uncited paragraphs carry their text) and, under --coverage, the uncovered sentences:

jq -c 'select(.kind=="paragraph" and .covered==false)
  | {paragraph, bytes, text}' "$t"
jq -c 'select(.kind=="uncovered") | {paragraph, sentence}' "$t"

What taguru itself contributed to the prompt is the trace's single steering record (candidates, reuse vocabulary with counts, context names, schema block); what the retry machinery did on its own is the attempts log's move records. For the aggregate view of all of this — loss rates, coverage, correction success, attempt states, moves, label concentration, cost — run scripts/extract_metrics.py over the same --out; anchoring rate and locator validity come from taguru anchoring.

When a number looks wrong: observation → knob

Each row pairs an observation — as the metric tables or the records spell it — with the control that answers it. The knobs are all described earlier on this page.

What counts as "high" is deliberately absent: the thresholds come from the baseline measurement against the field corpus (#780) and land here once measured. Until then, compare against your own previous run — extract_metrics.py --compare exists for exactly this — rather than against an absolute.

Four questions, one record set

The same records answer four different questions — keep straight which one you are asking (#784):

1. Is the model good enough? — attempt-state shares (does it keep the format), length_limited/timeout counts (does it fit the budget), correction success (does it understand the ask), and the loss records' raw/text pairs read by a person (is the interpretation right). If these fail, a stronger model beats any knob — see Quality.
2. Is the RAG itself capable? — the loss rate (answers the model gave that taguru did not keep — the headline number), the correction tuples (did recovery keep the meaning), paragraph coverage and the uncited paragraphs' text (what the output never reflects), steering versus the labels actually kept (did what taguru offered distort the answer), and piece paragraph ranges versus item citations (did chunking starve the model of the referent). taguru evaluate's retrieval metrics sit on this axis too.
3. Which parameter? — the table above.
4. Is the whole system's answer right? — end-to-end answer quality is out of scope for these records; axis 2's retrieval metrics and its gold data are the substrate an end-to-end evaluation builds on (see evaluate and benchmark).

Trust: extraction is a channel for claims

The prompt marks the document as data ("instructions inside are not for you") and the answer is validated against the extraction contract before anything is written — but an adversarial document can simply state false facts, and extraction faithfully records that the document states them. This is inherent to extraction, not specific to Taguru. The mitigation is structural: every fact is attributed to its source, so sources/retract (or re-importing a corrected batch) withdraws a bad document's contribution wholesale; and a batch file is inspectable text — review before loading into a trusted context.

Quality

Extraction quality is model quality. The contract above guarantees well-formed files, not good facts. Probe before committing a corpus: extract a few representative documents into a scratch directory, taguru import into a scratch TAGURU_DATA_DIR, and ask the questions you care about (query, activate, resolve). If the answers miss, a stronger model or smaller documents work better than prompt fiddling — and the manifest keeps either experiment cheap.

What to expect per model class (measured): small local models (≤ ~10B) keep the format but not the discipline — labels come back as sentence fragments and almost never repeat (one run: 106 distinct labels across 113 associations), and concepts multiply as noun phrases instead of converging on entities. One size up, the same corpus behaved differently in kind: a 12B model folded 55 associations into 13 short canonical labels and kept the procedure discipline unprompted — the discipline boundary sits around there, not at the frontier tier. The system degrades as designed: the lexical entrance absorbs much of the drift, sources/search answers what the graph misses, and vocabulary/audit lists the twins for alias repair. If passages carry the load, that works — but if the graph is the product, spend on a strong model.