Retrieval & citation quality gate — taguru evaluate

The CLI shape

taguru evaluate --eval eval.jsonl --context sake \
  --thresholds thresholds.json --out evaluation.json

--eval FILE          eval.jsonl (ADR 0003 §11's shared dataset, read under
                    this verb's own #215 extension fields — see below)
--context NAME       the already-populated context to evaluate
--url URL            the server to query; default resolves the same way
                    `taguru health` does (TAGURU_ADDR, or --config/
                    TAGURU_CONFIG)
--config FILE        load before resolving --url (same as --config
                    everywhere else)
--out FILE           where to write the artifact (default evaluation.json)
--thresholds FILE     a checked-in JSON file of regression bounds (see
                    below); a completed run that violates one
                    exits 3. Without it every completed run exits 0 and is
                    report-only — a stderr line says so, so a CI job that
                    forgot the flag does not pass silently.
--assembly           swap the passage lane for POST /contexts/{name}/evidence
                    (see evidence assembly) — the
                    structural lane itself never changes, so a baseline/
                    assembly run pair stays comparable on coverage and on
                    lanes.structural_hit specifically. lanes.passage_hit is
                    NOT identical across modes: assembly's hits[] mixes
                    passage/community locators with graph associations'
                    own citation_refs into one ranked pool, so a case that
                    only structural evidence answers can read passage_hit:
                    true under --assembly where baseline reads it false.
                    Without --assembly, behavior is unchanged from before
                    this flag existed.
--max-items N
--max-bytes N        equal-budget ceilings applied to BOTH modes — see
--max-tokens N        proving assembly
                    helps at equal budget. None given leaves baseline
                    untruncated, matching every run before this flag
                    existed — but --assembly always sends a budget to
                    POST /contexts/{name}/evidence (that endpoint has no
                    unbudgeted mode), so an --assembly run with no flag
                    still runs under the server's own defaults
                    (max_items 40, max_bytes 65536, max_tokens 4000).
--rerank MODEL        opts an --assembly run into a configured reranker;
                    usage error without --assembly.

This is a hand-rolled parser, like every other verb in this tree — no argument-parsing library. Because the default mode has no subcommand name of its own, evaluate has one rule the others don't: a leading argument starting with -- selects the default (run) mode; a leading bare word selects a subcommand — today, only compare. --eval and --context are required; every other flag is optional. Positional arguments are refused in this mode. Exit codes:

A CI job needs to tell "the corpus regressed, read the report" (3) apart from "the run itself failed, retry" (1) and from "you passed a bad flag" (2) — three different remediations, one exit code apiece. Without --thresholds, a completed run is always 0 regardless of what the metrics say; evaluate prints one stderr line noting the run is report-only so that omission is never silent.

What this is not

Five neighbors that measure something adjacent, and never a verdict

evaluate is the only one of the following that produces a pass/fail judgment. Everything else in this list either measures one thing in isolation or is explicitly forbidden from ranking, scoring, or picking a winner.

calibrate: one model's semantic floor, not retrieval quality

taguru calibrate --context NAME --probes FILE measures the semantic-floor bands of a running server's embedding model and suggests a TAGURU_SEMANTIC_FLOOR. That floor is a property of the model, not of the corpus or the question set — a probe file is cue<TAB>expected pairs where the cue is a spelling-free paraphrase, chosen precisely so the measurement stays independent of any one corpus's content. It writes nothing to disk; the report is stdout only (or --json), and even an overlapping band — "this model cannot separate these two names at this dimension" — is exit 0, because an honest overlap finding is calibrate's success, not its failure. An overlap can show up on an otherwise healthy corpus and a correctly-ranking model too — short concept-name glosses (a few characters, common for entity names in Japanese) leave little text for a model to anchor a separation on; see Troubleshooting — the semantic lane for a measured case. evaluate reads whatever floor calibrate suggested (via eval.jsonl's options.floor, or the server's own configured default) and measures what that floor, applied to a labeled question set, actually retrieves.

explain_resolve / explain_search: one query, always 200

POST /contexts/{name}/resolve/explain, /resolve_label/explain, and /sources/search/explain — exposed to an LLM client as the explain_resolve, explain_resolve_label, and explain_search MCP tools — diagnose exactly one cue against one expected name, or one query against one source, for a human or agent debugging a single miss. Every verdict is a 200: not_in_vocabulary, below_floor, below_cutoff, and served are equally valid answers, because a diagnosed miss is the endpoint's success, not its failure. There is no dataset, no aggregation, and no CLI surface at all — these are single-call diagnostics, reachable over HTTP, MCP, or the SDK. evaluate never calls them; it runs the same resolve/resolve_label/sources/search endpoints these explain routes diagnose, across an entire labeled dataset, and turns the aggregate into a gate. When evaluate reports a coverage or recall miss, reaching for explain_resolve/explain_search on that one case is exactly how you find out why.

The ANN recall guard: synthetic vectors, not your corpus

Once one context's passage vectors cross a compiled-in row threshold, the semantic lane switches from an exact cosine sweep to a small IVF (inverted-file) index for speed — a microbenchmark decision, not a knob. Its regression guard lives entirely inside cargo test (src/embedding.rs): a deterministic synthetic vector store, a brute-force top-k as ground truth, and an assertion that the index's own top-k overlaps it by at least 80% — "still high recall," never a brittle exact pin. A second, #[ignore]d test compares wall-clock speed at scale, and a third, also #[ignore]d, is the calibration sweep the threshold itself was picked from. None of the three touch a real corpus, a labeled question, or a citation — they guard one internal data structure's approximation quality against synthetic data. evaluate's recall.recall_at_k/mrr/ndcg measure something categorically different: whether your corpus, searched with your configuration, answers your labeled questions — the vector index underneath might be exact or approximate, and evaluate does not know or care which.

benchmark extract / compare (#189): extraction, not retrieval

taguru benchmark extract runs one taguru extract subprocess per (model, run) cell across a model matrix, and taguru benchmark compare is a pure, network-free function of that results directory measuring extraction throughput, volume, and run-to-run stability. Nothing here ever populates a live context or calls a search endpoint — the whole pipeline is about what a model extracts from documents, never about what a populated context retrieves. No single score or ranking is possible by construction: a mechanical test asserts the emitted key set never contains rank/score/winner/best/ recommended/overall/delta_vs_*. evaluate shares no artifact and no code path with this pair at all — it starts from an already-populated context and never touches an extraction model or a subprocess.

benchmark search (#260): model A's corpus vs. model B's, not today vs. yesterday

taguru benchmark search builds one per-model corpus from a benchmark extract results directory and searches the same question set against every one of them, reporting only differences between models' result sets — gold-data-free and judgment-free by construction, exactly like compare above. evaluate and benchmark search share exactly one artifact — eval.jsonl — and nothing else: benchmark search reads it in carry-through mode (every #215 extension field rides through opaque and unvalidated, earning one warning per run if seen), while evaluate reads it in interpret mode (typed, validated, no warning). The two verbs answer different questions with the same file: benchmark search asks does model A's corpus retrieve differently from model B's, holding the question set fixed; evaluate asks does this one corpus, today, still answer its labeled questions the way it used to — a quality gate over one corpus across time, not a comparison across corpora. See the dataset section on benchmark.html for the fields benchmark search itself reads.

Execution model: three lanes, fixed order, no fusion

Per case, up to three independent lanes run in a fixed order. No lane's result ever feeds another, and no lane's raw score is folded into another lane's — recall/coverage scoring reads only rank and label, never a hit's or a resolution's own score field.

Structural lane: coverage cues, then association probes

Coverage. For each of the case's cues[] (falling back to query when empty), evaluate calls /resolve and /resolve_label with an explicit limit of 5 — omitting it means "the ceiling itself," up to 1000 candidates, which no case needs. Because tier scores are not comparable across tiers, only the highest tier present is expanded — a lexical candidate list and a semantic one are never mixed. resolved_names[], resolve_tier, and the limit used are recorded per cue, so a structural miss is diagnosable as "resolution failed to find the name" versus "the graph has no such edge."

Association probes. For each expected_associations[] entry, evaluate resolves subject and object via /resolve and label via /resolve_label, then calls /query with all three positions pinned — but only if resolution yields exactly one candidate in the highest tier present for every position. Zero candidates is not_found; two or more same-tier candidates is ambiguous; in either case /query is never called for that entry, and no combination is guessed at or fanned out over. The per-entry outcome is recorded, so an ambiguous expectation reads as a visible, reproducible diagnostic — "write a more specific cue" — never a silent skip.

Why /query, not /recall

Context::recall(cue) returns every edge incident on a concept, and the HTTP layer pages it at clamp(limit, 100, 1000) — a hub concept with more than 1000 incident edges can push the expected triple outside the page, a silent false miss that looks like a retrieval failure but is a paging artifact. /query pins the exact triple instead, returns total, and its cost does not grow with the subject's degree.

activate, explore, and describe are never called

activate's and explore's results depend on decay/max_depth, which no eval field declares — there is no principled way to choose those parameters for a case that never asked for them. describe is equally tempting to reach for and is named here explicitly so a future reader does not assume it was overlooked rather than excluded. "Passage fallback" names no server behavior either — no lane fuses into or falls back to the other. What evaluate reports instead is a lane cross-tab (lanes.structural_hit/passage_hit/both/ neither), computed only over cases that declare both a structural and a source expectation and whose passage lane completed — a failed passage lane cannot honestly report passage_hit either way, so it is excluded from the denominator just like it is excluded from recall.

eval.jsonl: the dataset shared with benchmark search

{"taguru_eval":1,"name":"sake retrieval cases","default_target":{"context":"sake"}}
{"case_id":"brand-origin-001","query":"青嶺はどこの蔵の酒か","cues":["青嶺"],
 "expected_sources":[{"source":"corpus/brewery.md","paragraphs":[0],"relevance":3}],
 "expected_concepts":["青嶺酒造"],
 "expected_citations":[{"source":"corpus/brewery.md","paragraph":0,"section":"沿革",
   "quote":"1897年に創業"}],
 "options":{"limit":10}}

One taguru_eval header (equality-checked — this file is hand-authored, so a version this build was not built for is refused rather than silently reinterpreted), then one case per line. This is the exact dataset taguru benchmark search reads — the one artifact the two verbs share, and nothing else. evaluate loads it in interpret mode: every field this page documents is typed and validated, and a malformed value is a reported parse error like any other field (exit 2) — no warning, because nothing here is an unread extension for evaluate. benchmark search loads the same file in carry-through mode: every field below rides through opaque and unvalidated, earning one warning per run (never once per case) if seen.

options.sources is retired as a documented key: it never named a real server field (sources/search's own filter is by tag, not by source id), and no known dataset ever used it, so it is simply absent from the schema — an unknown-field rejection catches it in either mode.

expected_citations[].section: three-valued, not a serde artifact

An absent section key means "don't check section." An explicit "section": null means "assert this paragraph is outside every stored section" — a real, checkable claim, since the server's own citation locator never omits the key on the wire. A string value is compared as-is. quote, when given, is normalized-substring matched against the citation's own paragraph text — since a citation's text is exactly one paragraph, a quote spanning a paragraph boundary can never match; split it into two expected_citations entries instead.

Metric catalog

Every metric below is computed purely from what the three lanes already returned — no lane is ever called twice, and scoring never reads a hit's or a resolution's own score/cosine field, only rank and label. Percentiles inside a Distribution are nearest-rank, matching every other artifact in this tree.

Citation recall and locator validity are never merged into one score — they answer different questions (did search surface the evidence, versus is the locator itself correct) and a case can score high on one while scoring low on the other. Matching against expected_concepts/expected_labels uses taguru::context::normalize_entry — the same folding the passage index itself applies — never benchmark::identity::normalize_term, whose deliberate katakana exception exists for a cross-model comparison this verb does not do (the corpus is fixed here; no model is being compared). expected_sources matches by exact (source, paragraph) instead — a paragraph index is not text to normalize. Both choices are recorded verbatim in evaluation.json's own matching block.

evaluation.json

{
  "taguru_evaluation": 1,
  "generated_at": "2026-07-28T10:05:12Z",
  "matching": { "normalization": "taguru::context::normalize_entry",
      "normalized": ["expected_concepts", "expected_labels"],
      "sources": "exact (source, paragraph) match — no normalization" },
  "inputs": { "eval": { "path": "eval.jsonl", "name": "sake retrieval cases", "cases": 12 },
      "context": "sake", "url": "http://localhost:8248", "out": "evaluation.json",
      "default_limit": 10, "resolve_limit": 5,
      "mode": "assembly",
      "budget": {"max_items": 40, "max_bytes": 65536, "max_tokens": 4000},
      "rerank": "bge-reranker-v2-m3" },
  "corpus": { "revision_before": { "...": "..." }, "revision_after": { "...": "..." },
      "stable": true, "last_write_epoch_before": 1234, "last_write_epoch_after": 1234,
      "embeddings": { "provider_model": null }, "sources_count": 5 },
  "thresholds": null,
  "definitions": { "recall.recall_at_k": { "unit": "ratio", "statistic": "distribution",
      "scopes": ["case"], "description": "...", "source": "...", "caveat": null }, "...": "..." },
  "warnings": [],
  "cases": [ { "case_id": "brand-origin-001", "query": "青嶺はどこの蔵の酒か", "limit": 10,
      "passage": { "outcome": "searched", "plan": { "...": "..." }, "hits": [ { "source": "corpus/brewery.md",
        "paragraph": 0, "score": 0.91, "lanes": { "...": "..." } } ], "latency_ms": 4 },
      "recall": { "recall_at_k": 1.0, "mrr": 1.0, "ndcg": 1.0, "expected_total": 1, "matched": 1 },
      "coverage": { "concepts": { "expected": 1, "matched": 1, "value": 1.0 } },
      "citations": { "recall": { "expected_total": 1, "matched": 1, "value": 1.0 },
        "validity": { "expected_total": 1, "valid": 1, "value": 1.0 },
        "checks": [ { "source": "corpus/brewery.md", "paragraph": 0, "served": true,
          "outcome": "resolved", "section": { "check": "matched", "expected": "沿革" },
          "quote": { "declared": "1897年に創業", "matched": true }, "latency_ms": 6 } ] },
      "missed": [], "missed_truncated": 0,
      "evidence": { "latency_ms": 4, "items": [ { "candidate_id": "passage\u0000sake\u0000corpus/brewery.md\u00000",
          "kind": "passage", "fused_rank": 1, "source": "corpus/brewery.md", "paragraph": 0,
          "citation_refs": [] } ],
        "omitted_by_reason": {},
        "selection": { "dedup_dropped": 0, "contradiction_groups": 0, "diversity_tier_width": 10 },
        "reranker": { "configured": true, "ran": true, "model": "bge-reranker-v2-m3" } },
      "budget": { "items_used": 2, "bytes_used": 612, "tokens_used": 158,
        "limits": {"max_items": 40, "max_bytes": 65536, "max_tokens": 4000}, "omitted_total": 0 },
      "diversity_sources": 1 } ],
  "metrics": { "recall.recall_at_k": { "n": 12, "min": 0.5, "p50": 1.0, "p90": 1.0, "p99": 1.0,
      "max": 1.0, "mean": 0.93, "sum": 11.2 }, "...": "..." }
}

inputs.mode/.budget/.rerank and the per-case evidence/budget/diversity_sources blocks (#308, ADR 0006 §14) are additive — taguru_evaluation stays 1. inputs.budget is absent in baseline mode unless a --max-* flag was given; inputs.rerank is absent unless --rerank MODEL was. A case's own evidence block is present only in --assembly mode; budget is present in either mode once a budget flag applies. See evidence assembly for the full EvidencePackage shape evidence.items[] mirrors (stripped of corpus body text, ADR 0004 §11).

Stamp taguru_evaluation: 1, range acceptance — taguru writes this file and its own compare mode re-reads it, so within an accepted range a later revision may only add a field, never remove or repurpose one. Every field is #[serde(default)]. A metric's concrete shape — Distribution, Ratio, or Count — is read by first loading definitions[metric].statistic and deserializing that metric's value into the type its own definition names, never through the value type's own untagged deserialization directly: a Ratio's {"value":0.09,"n":31, "numerator":3} would otherwise succeed silently as a Distribution{n:31, min:None, …}, since only n is a Distribution's required field. Reading through definitions first turns a metric with no matching definitions entry into a loud failure instead of a silent one.

Without a budget flag, hits[] is not capped below a case's own options.limit — the bound is exactly that case's configured limit, so an external offline scorer can build a recall@k for any k up to it (a case wanting an offline recall@50 must set that case's own options.limit to 50 or higher). Two things narrow this, both by design: a --max-items/--max-bytes/--max-tokens flag can truncate hits[] below options.limit in either mode (that is the whole point of an equal-budget comparison — see evidence assembly's budget semantics); and in --assembly mode, hits[] is capped at options.limit rather than left to the admitted package's own item count (up to budget.max_items, default 40 — structurally unrelated to options.limit), so a baseline/assembly run pair at the same --limit scores over the same k. No corpus body text is ever written into this artifact; on a quote mismatch, only the user's own declared quote and a boolean match result are recorded, never the served paragraph body. missed[] is capped at 3 entries with a missed_truncated count of what was dropped (not the total). The resolved server URL is recorded as scheme + host + port only — never with userinfo, and never the bearer token in any form.

Corpus revision bracketing

corpus.revision_before/revision_after are read once before the first case and once after the last; stable is equality across all three revision lanes (graph, passages, config), never ordering. A changed revision does not abort the run — aborting a multi-minute CI run because one unrelated document was imported mid-run is worse than finishing and flagging it — but an unstable run fails the gate by default once --thresholds is given; opting out requires "allow_unstable_corpus": true in the thresholds file. embeddings. provider_model records the server's configured embedding model name (never a URL or key) so a run's semantic-lane identity is reproducible without re-querying the server.

No answer-generation LLM is required, and this is enforced, not merely claimed: a source-level test asserts src/evaluate.rs and its submodules contain no TAGURU_EXTRACT_/TAGURU_EMBED_ string literal and no crate::extract/crate::embedding import, and the offline fixture suite asserts the server's own vector_off_reason string appears verbatim when no provider is configured — proving the server, not just this process, had none. evaluate itself never calls an LLM, but the server it queries may embed the query text for the vector lane — that is server-side configuration, recorded via embeddings.provider_model, not a prohibition on the server having a provider at all.

The thresholds file and the exit-3 gate

{
  "taguru_evaluate_thresholds": 1,
  "aggregate": { "recall.recall_at_k": { "min": 0.8 }, "citations.recall": { "min": 0.9 } },
  "cases": { "default": {}, "overrides": { "known-miss-003": { "citations.recall": { "min": 0.0 } } } },
  "allow_unstable_corpus": false
}

--thresholds FILE is a checked-in, user-authored JSON file — stamp taguru_evaluate_thresholds: 1, equality-checked (the same posture eval.jsonl's own header takes, not evaluation.json's range acceptance, because a human authors this file by hand), and deny_unknown_fields. Three top-level keys:

A bound needs at least one of min/max, both finite, and min <= max when both are given. Only ten metric names are legal inside cases.* — recall.recall_at_k, .mrr, .ndcg, coverage.concepts, .labels, .associations, citations.recall, .locator_validity, latency.passage_ms, and diversity.sources (#308, ADR 0006 §14 — the one --assembly-era metric compared case-by-case; budget.*/rerank.ran stay aggregate-only). The three per-call latencies (resolve_ms/query_ms/citation_ms) are deliberately excluded: a single case can make more than one resolve/query/citation call, so there is no one per-case value to bound. deny_unknown_fields cannot catch an unknown metric name inside aggregate, or an unknown case_id inside cases.overrides — both are checked explicitly, as a post-load set difference against evaluation.json's own definitions keys and the loaded eval.jsonl's case_ids, and every problem found is reported together rather than just the first.

A metric a case never computed (it did not declare the matching expectation, or that lane never ran) is skipped, never violated. The threshold file's own byte content is hashed and recorded in evaluation.json as a threshold identity — this is what evaluate compare's mismatch warning checks between two runs, without needing either original file on disk at compare time.

taguru evaluate compare

taguru evaluate compare base.json head.json --out changes.jsonl

Reads two evaluation.json runs and classifies every case whose outcome moved between them. This is the one place in the tree whose whole job is to say which run was better — verdict vocabulary (improved, regressed) is legitimate here, unlike inside benchmark's artifacts. Exit codes: 0 comparison completed (regardless of regressions found — the pass/fail gate is evaluate --thresholds, not this subcommand) · 1 changes.jsonl could not be written · 2 usage error, an unreadable file, or malformed/out-of-range JSON.

{"kind":"header","taguru_evaluation_changes":1,"generated_at":"...",
 "base":{"path":"base.json","context":"sake","taguru_evaluation":1,"...":"..."},
 "head":{"path":"head.json","context":"sake","taguru_evaluation":1,"...":"..."},
 "counts":{"improved":2,"regressed":1,"added":0,"removed":0,"unchanged":9,
   "cases_base":12,"cases_head":12},
 "metrics":[{"metric":"recall.recall_at_k","statistic":"distribution",
   "sides":{"base":0.83,"head":0.90},"delta":0.07}]}
{"kind":"regressed","case_id":"brand-origin-004",
 "metrics":[{"metric":"citations.recall","sides":{"base":1.0,"head":0.5},"delta":-0.5,
   "direction":"regressed"}]}

Line 1 is always the header; every line after is one record per changed case only — kind ∈ improved | regressed | added | removed. Unchanged cases are counted in the header's counts.unchanged, never emitted per case, keeping the file's size bounded by the number of cases whose outcome actually moved. Classification reads only nine of the ten --thresholds-scoped metrics — every one except latency.passage_ms, whose single-run noise would flag most cases on a mere rerun rather than on a genuine quality change. A case with one metric down and another up is classified regressed, quality-gate-first, and the record still lists both directions. added/removed cases carry one-sided metrics (direction: "not_comparable"). A mismatched context, corpus revision, threshold-file identity, or taguru_evaluation stamp between the two files is a warning, never a refusal — and so is a mismatched inputs.budget between the two runs (#308, ADR 0006 §14): comparing citation recall or source diversity across different budgets is exactly the dishonest comparison an equal-budget evaluation exists to catch.

What never lands in these artifacts

No corpus body text, ever — the one bounded exception is a quote mismatch, where the user's own declared quote and a boolean result are recorded, never the served paragraph. The resolved server URL, scheme + host + port only, never userinfo or a bearer token. Server error messages truncated to 200 bytes on a UTF-8 character boundary. missed[] capped at 3 entries per case, with the dropped count recorded separately. hits[] is not capped below a case's own options.limit when no budget flag applies — a HitLocator is roughly 40 bytes, so even a limit of 1000 is cheap, and capping below the case's own configured limit would prevent an external scorer from ever seeing the full window that case asked for. A --max-items/--max-bytes/--max-tokens flag is the one deliberate exception (see above).