Extraction benchmark — taguru benchmark extract
Runs taguru extract across a matrix of models,
one taguru extract --diagnostics-out subprocess per (model, run) cell, every
cell over the same corpus under the same task settings. Writes a reproducible
results directory taguru benchmark compare — or any
external tool — can derive measurements from without spending model time again, and
taguru benchmark search can build a per-model corpus from
and compare its live search results against. The design is fixed by
ADR 0003;
this page documents the shape it produces.
The CLI shape
taguru benchmark extract --models models.json --context bench \
--out results/ --runs 3 corpus/
--models FILE model matrix (schema below)
--context NAME the context every cell's batch files target
--out DIR results directory; re-running the same --out resumes —
a cell already recorded complete or failed is never re-run
--runs N runs per model, 1-99 (default 1)
--questions N forwarded to every cell's --questions
--fact-budget N forwarded to every cell's --fact-budget
--no-passage forwarded to every cell's --no-passage
--candidates forwarded to every cell's --candidates (ADR 0014)
--vocabulary PATH forwarded to every cell's --vocabulary (ADR 0015)
--lossy forwarded to every cell's --lossy
--description TEXT forwarded to every cell's --description
--parallel N forwarded to every cell's --parallel (default 1)
--max-output-tokens N forwarded to every cell's --max-output-tokens
--max-attempts N forwarded to every cell's TAGURU_EXTRACT_MAX_ATTEMPTS,
1-10 (default 2)
CORPUS_DIR exactly one directory (.md/.txt, sorted by name) — every
cell sees the same document order, since a document's
relation-label vocabulary is offered to the next
document's prompt (see chunking)
Every one of these settings is global to the matrix, never per model — the fairness
invariant below is why. Only what a provider is or can do — its endpoint,
credential, name, and structured-output rung — lives in models.json. Exit
codes match taguru extract's own convention: 0 every cell
completed · 1 a harness failure (a cell's own document failures still count as
0 at this level — see cell outcomes below) ·
2 usage error · 130 stopped (Ctrl+C), resumable.
One binary, one path
A cell runs the exact taguru extract binary an operator runs, selected the
exact same way (TAGURU_EXTRACT_* in the child's environment) — a measurement is
evidence about the product, not about a benchmark harness's own reimplementation of it. The
child is spawned via std::env::current_exe(), so it is always this same build.
The cell owns its environment: before every spawn, every TAGURU_EXTRACT_*
variable is removed from the inherited environment, then the cell's own resolved values are
set explicitly — including ones left at their defaults. A stray
TAGURU_EXTRACT_LOSSY=1 in the operator's own shell cannot silently make one
cell unfair, and no TAGURU_EXTRACT_* value a cell runs under is "whatever
happened to be inherited." This scrub is scoped to that one namespace — proxy, locale, and
TLS-trust variables pass through unscrubbed, since extract itself reads none of
them.
Every cell gets a fresh cells/<model_id>/run<NN>/ directory — never
--force, which would discard extract's own chunk-checkpoint resume
(its checkpoints guide). Two cells never share a
directory, so extract's own manifest/checkpoint skip logic — which does not key
on endpoint URL, timeout, or --parallel — cannot cross-skip between them even
when two models happen to share a wire name.
models.json: the model matrix
{
"taguru_benchmark_models": 1,
"defaults": {
"timeout_secs": 300,
"structured_output": "auto"
},
"models": [
{
"id": "qwen25-7b-q4",
"label": "Qwen2.5 7B Instruct (Q4_K_M, local Ollama)",
"model": "qwen2.5:7b",
"url": "http://localhost:11434/v1/chat/completions",
"api_key_env": null,
"structured_output": "auto",
"timeout_secs": 300,
"note": "baseline"
},
{
"id": "hosted-120b",
"label": "gpt-oss-120b (hosted, OpenAI-compatible)",
"model": "gpt-oss-120b",
"url": "https://example.internal/v1/chat/completions",
"api_key_env": "BENCH_KEY_HOSTED",
"structured_output": "json-schema"
}
]
}
taguru_benchmark_models must equal 1 exactly — unlike
manifest.json below, this file is authored by hand, not written and re-read by
taguru itself, so a shape this build was not built for is refused rather than silently
defaulted. id matches ^[a-z0-9][a-z0-9._-]{0,63}$, must be unique,
and is the only value used as a path component (model names carry ://
that are not portable, e.g. qwen2.5:7b). defaults supplies any key
an entry omits; the resolved record (defaults folded in) is what models.lock.json
and manifest.json record, so no reader has to re-apply defaults. An unknown
top-level or per-model key earns a warning naming it, never a hard failure — a hand-edited
file is exactly where a misspelled knob silently becomes a no-op.
The fairness invariant
A per-model record may describe only what the provider is or can do. Everything
that shapes the task — the corpus, --context, --questions,
--fact-budget, --no-passage, --lossy,
--description, --parallel, --max-output-tokens,
--max-attempts, the run count — is global to the matrix, a flag on
taguru benchmark extract, never a per-model key. Stated as an invariant, not a
convention: models.json has no field where a task setting could even go, so it
cannot encode a comparison measurements.json would show as one model simply
being more productive when it was really just given a larger budget.
structured_output is the one deliberate per-model exception — the rung a
backend can actually honor is a capability, not a task setting, and auto probes
the live endpoint per cell regardless of what the file says.
Secrets
No API key value ever appears in models.json, in models.lock.json,
or in any artifact under the results directory. api_key_env names an environment
variable; the harness reads it in its own process and passes the value to the child's
environment only, never to a file. A named variable that is unset is a usage error at
startup, before any model is called. A models.json carrying a key-shaped value
where api_key_env belongs, or a url whose authority carries inline
user:password@ userinfo, is a hard usage error at parse time — naming the
correct field and refusing to run any cell — rather than a value silently redacted on the
way out.
Results directory layout
<out>/
manifest.json # reproduction record, document/chunk dictionary
models.lock.json # models.json fully resolved, no secrets
runs/<model_id>.run<NN>.jsonl # one file per cell, the AttemptRecord superset
cells/<model_id>/run<NN>/ # passed verbatim as `taguru extract --out`
.extract-manifest.json # written by extract, untouched by the harness
.extract-checkpoints/ # written by extract
.extract-trace/ # written by extract (ADR 0023)
<batch files> # raw extraction output
diagnostics.jsonl # passed as --diagnostics-out
stdout.log stderr.log exit_code # the cell's own record
measurements.json measurements.csv # written by `taguru benchmark compare` (below)
differences.jsonl # written by `taguru benchmark compare` (below)
run<NN> is zero-padded (run01, run12) so lexical
order is run order. Nothing outside cells/ is written by a child process, and
nothing inside cells/ is edited by the harness — everything downstream of the
models (taguru benchmark compare, or any external tool)
is a pure function of the results directory alone.
manifest.json
Written once after preflight (models validated, corpus enumerated and hashed, every model
probed) and updated after every cell concludes. Carries: run_id,
started_at/finished_at (the latter null until every
cell in the matrix has run), taguru_version, harness (execution
mode, runs per model, the corpus root, the resolved document order, the models file's path
and hash), extraction_settings (every task setting above, plus
prompt_version/chunk_bytes and a hash of the canonical structured-
output schema), the documents[] dictionary (one entry per corpus document: its
id, path, byte length, content hash, paragraph count, and its chunk-by-chunk provenance —
hash and paragraph range per chunk, computed once here and reused verbatim by every cell),
models[] (each model's provider-probe facts — digest/quantization/context
window when an Ollama-shaped /api/show answers, absent with a note for a plain
OpenAI-compatible endpoint), and cells[] (one entry per concluded cell: its
directory, its resolved structured-output rung, and its outcome).
A probe-dependent field (a model's digest, an environment's CPU model) is present and
null when unavailable, never omitted — a reader can tell "asked, nothing came
back" from "this build never records such a field." The document dictionary lives here and
only here: chunking depends on the document and chunk_bytes alone, never on the
model, so repeating it per cell would be N×M copies of one fact.
taguru_benchmark_manifest accepts any version from 1 through the
current one — unlike models.json, this file is written and later re-read
by taguru itself (on resume), and compatibility across the tool's own evolution is the point
of keeping one on disk.
runs/<model_id>.run<NN>.jsonl
One file per cell — the cell is the unit of execution, failure, and resume. Every line is a
tagged JSON object; kind discriminates:
header— line 1 only: version stamp,run_id, cell identity.document— two per document attempted:phase: "start"(identity only, written the moment the document's first record is seen) andphase: "end"(counts once it finishes —outcome: "written"fromextract's own successful-completion record, or a synthesizedoutcome: "failed"with every countnullwhen the cell concludes with that document still open). Astartwith no matchingendmarks a document abandoned mid-cell — the same signal one level down fromkind: "cell"'s own absence.chunk— one per chunk,extract's own provenance record (chunk_sha256, paragraph_first/paragraph_last) carried through with a harness envelope added.attempt— one per completion call, built from three layers: everyAttemptRecordfield extract's own sidecar would have written, unrenamed and unmodified; the harness envelope (ts,cell_id,model_id,run_index,document_id); and, denormalized from the document dictionary,document_sha256/chunk_sha256/paragraph_first/paragraph_last— so any single line is dereferenceable without a cross-kindjoin.tsis the harness's own observation time (the sidecar carries no wall clock); an attempt's true duration iselapsed_seconds.cell— the last line, written only when the cell concludes cleanly (exit code 0 or 1): totals and the cell's outcome. Its absence marks an interrupted cell — the top-level "was this cell actually finished" signal.
Every consumer joins by key — (document_id, chunk_index) — never by line
position: --parallel > 1 interleaves a document's own chunk lines, and a
resumed cell's retry segment is appended after whatever the interrupted attempt already
wrote.
Cell outcomes and resume
A cell's outcome comes straight from its child's exit code: 0 →
complete, 1 → failed (the cell ran the whole corpus;
some documents did not extract — that failure rate is itself a measurement, not a
harness error), 130 or a signal → interrupted. complete
and failed are both terminal: re-running the same --out skips
every cell already recorded as either, so a rerun costs nothing for work already done.
An interrupted cell is retried on the next invocation, into the exact same
cells/<model_id>/run<NN>/ directory, never with --force
— extract's own .extract-manifest.json/
.extract-checkpoints/ then resume at the document and chunk level exactly as
they would for a bare rerun of taguru extract (see
chunk checkpoints). The accepted cost: a document
that was in flight at the moment of interruption may show a second start record
on retry, and a chunk that was already checkpointed may repeat its own chunk
line — both harmless, since every join is by key, never by line position.
Resuming an existing results directory re-validates that nothing about the matrix definition
has drifted: the hash of models.json, every task setting, and the corpus'
document dictionary must all still match what manifest.json recorded when the
directory was created. A mismatch is a usage error naming what changed, pointing at a new
--out — two differently-configured runs must never share a directory
(the same reasoning behind giving every cell its own fresh directory in the first place).
taguru benchmark compare
taguru benchmark compare [--with-text] results/
Reads a finished results directory — manifest.json, runs/*.jsonl,
and the written cells/** batches — and writes measurements.json,
measurements.csv, and differences.jsonl
into it, from one shared read of the directory. A pure function of the results
directory: no model is called, no network is touched (ADR 0003's R4), so re-running it
costs nothing and always reflects exactly what is on disk right now — including a directory
that a resumed taguru benchmark extract has since grown more cells into. Exit
codes: 0 written · 1 the directory is missing, its
manifest.json/runs/*.jsonl could not be read, or (with
--with-text) a corpus file named by the manifest is unreadable or has changed
since the results directory was created · 2 usage error.
measurements.json
{
"taguru_benchmark_measurements": 1,
"run_id": "...",
"generated_at": "2026-07-26T10:05:12Z",
"percentile_method": "nearest-rank",
"matching": { "module": "benchmark::identity", "case_fold": true,
"unicode_normalization": "NFKC", "alias_expansion": "batch-local", "weight_tolerance": 0.0 },
"inputs": { "runs": ["runs/qwen25-7b-q4.run01.jsonl"], "cells": "cells/" },
"definitions": { "latency.attempt_seconds": { "unit": "second", "statistic": "distribution",
"scopes": ["cell", "model", "document"], "description": "...", "source": "...",
"caveat": null }, ... },
"cells": { "qwen25-7b-q4.run01": { "model_id": "qwen25-7b-q4", "run_index": 1,
"latency.attempt_seconds": { "n": 31, "min": 4.12, "p50": 12.44, "p90": 31.87,
"p99": 58.10, "max": 61.02, "mean": 15.23, "sum": 472.1 }, ... } },
"models": { "qwen25-7b-q4": { "...": "same metric keys, pooled over that model's cells" } },
"documents": { "qwen25-7b-q4": { "brewery": { "run01": { "...": "the same document's own
metrics at run granularity" } } } }
}
Every distribution metric is {n, min, p50, p90, p99, max, mean, sum}.
Percentiles are nearest-rank, never interpolated: for ascending
x[0..n-1], the pth percentile is x[⌈p/100·n⌉ − 1] —
always an observed value, exactly reproducible by an external re-aggregator reading the same
runs/*.jsonl. A metric with zero qualifying samples still emits its key, with
n: 0 and every statistic null — never omitted, so a reader can
tell "measured, zero samples" from "this metric does not apply here" by key presence alone.
A ratio metric ({value, n, numerator}) follows the same rule: n: 0
pairs with value: null and numerator: null, never a
divide-by-zero NaN and never a silently misleading 0.0.
runs/*.jsonl alone cannot supply the extraction-shape metrics that need a
batch's full item list (the association weight split, distinct subjects/relations, alias
orphans, out-of-range paragraph locators) — compare re-reads the
cells/** batch a written document's own kind: "document" record
names, classifying each line leniently rather than with taguru import's
fail-fast batch reader, since counting malformed lines requires reading past the first one.
measurements.json's own inputs block names both sources, so the
derivation stays auditable.
No single score or ranking
This is structural, not a convention compare merely follows: per-model and
per-cell results are keyed maps in lexicographic order, never arrays a position in could be
read as a rank; the emitted key set is asserted (by a unit test) to never contain
rank, score, winner, best,
recommended, overall, or delta_vs_*; and
measurements.csv is tidy — model_id is always a data
column, never a header column a model's name could sort by. There is simply no field
anywhere in either artifact to put a rank in.
measurements.csv
scope,model_id,run_index,document_id,metric,stat,value,unit,n
cell,qwen25-7b-q4,1,,latency.attempt_seconds,p50,12.44,second,31
model,qwen25-7b-q4,,,latency.attempt_seconds,p50,12.91,second,93
document,qwen25-7b-q4,1,brewery,extraction.associations,value,41,association,1
| Column | Meaning |
|---|---|
scope | cell | model | document |
model_id, run_index, document_id | the key that scope needs; blank where a scope has no such axis (a model row has no run_index) |
metric | a dotted name keying definitions |
stat | value | min | p50 | p90 | p99 | max | mean | sum | n | numerator — whichever fields that metric's shape has |
value | empty exactly where the corresponding JSON field is null |
unit, n | copied from definitions and from the metric's own sample size, so a spreadsheet never needs the JSON open to interpret a row |
measurements.csv is a value projection of measurements.json,
not a lossless flattening of it: every numeric field of every distribution and ratio becomes
one row, but definitions — unit, statistic, description, source,
caveat — and the inputs/run_id/generated_at
metadata stay JSON-only. A tool that needs a metric's caveat reads
measurements.json; the CSV is for spreadsheets and pandas/
sqlite. It carries no version stamp of its own — its version is
measurements.json's, and the harness writes the two atomically, one
stage-then-rename each, CSV first.
Metric catalog
This table mirrors measurements.json's own definitions block —
the artifact carries the authoritative, machine-readable copy; this is a human-readable
index into it, not a second source of truth.
| Metric family | Scopes | What it measures |
|---|---|---|
latency.attempt_seconds | cell, model, document | Wall time of one completion call. |
latency.chunk_seconds | cell, model, document | One chunk's total time across every retry (cross-chunk correction excluded). |
latency.document_wall_seconds | cell, model, document | A document's phase: "start" to phase: "end" span. |
latency.seconds_per_association | cell, model, document | A written document's total attempt time per association produced. |
throughput.output_tokens_per_second | cell, model, document | One attempt's output tokens over its elapsed time. |
tokens.input_per_attempt, tokens.output_per_attempt, tokens.total_per_attempt | cell, model, document | Provider-reported token counts per completion call. |
extraction.associations_per_1k_input_tokens | cell, model, document | Associations produced per 1,000 input tokens spent, retries included. |
attempt.state_rate.* | cell, model | Share of attempts in each of ADR 0001 §7's terminal states (stop_valid, stop_malformed, length_limited, empty, refusal, timeout, transport). |
attempt.length_limited_rate, attempt.retry_rate, attempt.parse_error_rate, attempt.validation_rejected_rate, attempt.provider_metadata_missing_rate | cell, model | Attempt-level failure/retry signals independent of, or complementary to, state. |
attempt.finish_reason_rate.<reason> | cell, model | Share of attempts reporting each provider-specific finish reason actually observed in this run. |
document.written_rate, document.failed_rate | cell, model | Share of started documents that reached each terminal outcome. |
cell.complete_rate | model | Share of a model's cells that completed, read from manifest.json. |
extraction.associations, .concepts, .labels, .questions, .duplicates, .dropped | cell, model, document | Per-document extraction volume, from kind: "document"'s own counts. |
extraction.weight_positive, .weight_negative | cell, model, document | Association lines with positive vs. negative weight. |
extraction.subjects_distinct, .relations_distinct | cell, model, document | Distinct subject/label values within a document's own batch. |
extraction.paragraph_attributed_rate | cell, model, document | Share of association lines carrying a paragraph locator. |
extraction.relation_reuse_ratio | cell, model, document | Share of association lines whose label was not that document's first use of it. |
extraction.alias_orphans, .paragraph_out_of_range, .batch_lines_invalid | cell, model, document | Batch-writer health checks: aliases with no matching canonical, locators past a document's paragraph count, and lines matching none of the batch format's known shapes. |
stability.run_pair_jaccard | model | Association-key overlap between two of a model's own runs, one sample per unordered run pair, restricted to documents both runs completed. |
stability.keys_distinct | model | Distinct association keys observed across every run a model completed at least one document in. |
stability.keys_in_all_runs_ratio, .keys_in_single_run_ratio | model | Among keys whose document completed in 2+ runs, the share present in every one of those runs vs. in exactly one. |
stability.key_presence_ratio | model | Per key (document completed in 2+ runs), the share of that document's completed runs the key appeared in. |
stability.polarity_variation_ratio, .weight_variation_ratio, .attribution_variation_ratio | model | Among keys observed in 2+ runs, the share whose weight sign, weight value (past matching.weight_tolerance), or paragraph locator set disagreed across runs. |
stability.alias_canonical_variation_ratio | model | Among (document, alias-kind, spelling) triples declared in 2+ completed runs, the share whose resolved canonical was not the same in every declaring run. |
run.associations_total, .elapsed_seconds_total, .documents_written | model | One run's own totals — one sample per run a model has a cell for, including a run that completed zero documents. |
Run-to-run stability and the matching block
stability.* (issue #258) measures how much a model's own extraction varied
across its N runs — model scope only, since a cell is a single run and
documents' own model → document → run_label shape has no slot for
a value that spans runs. Two association lines count as "the same association" once
normalized (case folding, Unicode NFKC) and resolved through that batch's own alias
declarations (alias_expansion: "batch-local" — only the declaring batch's
alias lines are consulted, the same conflict-first-wins and chain-following semantics the
live graph's own alias resolution uses) into one benchmark::identity
AssocKey. This same-ness core is deliberately independent of
crate::context's own entry normalization — it does not fold katakana to
hiragana, since a model's choice of script between runs is exactly the kind of variation
these metrics exist to observe, not something to fold away first.
The parameters that judgment was made under are recorded verbatim in
measurements.json's own top-level matching block, shown above —
so the artifact stays re-derivable without reading benchmark::identity's
source. A document only ever contributes to these metrics through a run in which it
completed (outcome: "written" and its batch was readable); a harness or
model completion failure is document.written_rate's concern, not folded into
extraction instability here. differences.jsonl's
model-pair diff reuses this same module and records the identical matching
block in its own header (ADR 0003 §9.4), so a same-ness judgment about "did two runs agree"
and "did two models agree" is always traceable to one parameter set.
Known limitations
Carried forward as caveat strings in measurements.json's own
definitions, not hidden: attempt.state_rate.stop_malformed
conflates a JSON syntax failure with a Stage 1 validation rejection —
attempt.validation_rejected_rate separates them (ADR 0001 §7). The legacy
(non-ladder) extraction path never reports length_limited or
refusal as a state, folding both into stop_valid
instead — attempt.length_limited_rate reads the flag directly and is not
subject to that gap. With --no-passage,
extraction.paragraph_attributed_rate is structurally 0, since
taguru extract's own batch writer strips every paragraph locator before writing
when there is no passage line left to locate into — this reflects the flag, not model
behavior.
differences.jsonl
A paired diff between every pair of models in the matrix (issue #259, ADR 0003 §9.4):
what both models extracted in common, what only one of them extracted, and where the two
disagreed on polarity, paragraph attribution, or an alias's resolved canonical. There is no
gold data anywhere in this pipeline, so differences.jsonl makes no correctness
judgment — it reports differences, never errors, omissions, or which side is right.
A mechanical lexicon test asserts no emitted key name or kind value ever
matches miss/error/wrong/incorrect/
fail/omit/expected/gold/
truth/false positive/recall/precision/
better/worse, on top of the rank-word
ban measurements.json already carries. Sides are named a/
b, bound to a model_id — never baseline/
candidate — and an absent side serializes null, never a count.
{"kind":"header","taguru_benchmark_differences":1,"run_id":"...",
"pairs":[{"pair_id":"hosted-120b__qwen25-7b-q4","a":"hosted-120b","b":"qwen25-7b-q4"}],
"matching":{"module":"benchmark::identity","case_fold":true,"unicode_normalization":"NFKC",
"alias_expansion":"batch-local","weight_tolerance":0.0},
"text_included":false}
{"kind":"association_shared","pair_id":"hosted-120b__qwen25-7b-q4",
"present_in":["hosted-120b","qwen25-7b-q4"],
"key":{"subject":"beer co","label":"brews","object":"lager"},
"sides":{"a":{"runs":[1,2,3],"n_present":3},"b":{"runs":[1,2],"n_present":2}},
"locator":{"document_id":"brewery","source":"corpus/brewery.md","document_sha256":"9c2e...",
"paragraph":1,"chunk_index":0,"chunk_sha256":"41ab...","text":null,"text_truncated":false}}
One file covers every model pair — each record carries its own pair_id, and the
header's pairs array plus matching block make the file
re-derivable without reading benchmark::identity's source: a difference set is
uninterpretable without knowing whether case folding, NFKC, and batch-local alias expansion
were applied before "present in one side only" was decided. The matching block
is byte-identical to measurements.json's own — the same
same-ness judgment underlies both.
Record kinds
| kind | Fires when |
|---|---|
document_coverage | Once per (pair, document) either side attempted. Marks whether a document is eligible for the record kinds below at all — a document only one side ever completed a run for is excluded from every one of them, since a harness or model completion failure is document.written_rate's fact, not an extraction difference. |
association_shared | An association key both sides observed in at least one completed run, for a document both sides completed. |
association_single_side | A key only one side ever observed, on a document both sides completed. |
polarity_difference | A shared key whose two sides' observed weight-sign sets share no element — e.g. side a always positive, side b always negative. Within-side sign disagreement across a model's own runs is stability.polarity_variation_ratio's concern, not this. |
attribution_difference | A shared key whose two sides' observed paragraph-locator sets (None counts as its own value, not "no signal") share no element. |
alias_resolution_difference | A (document, alias-kind, spelling) triple both sides declared in at least one completed run, whose resolved-canonical sets share no element. |
surface_form_variation | A key (shared or single-side) that two or more distinct raw subject/label/object spellings folded into — a normalization candidate, surfaced whether the variation is between the two models or entirely within one model's own runs. |
Every difference kind fires on set disjointness, never on inequality: two sides whose
observations merely overlap (side a attributed paragraph 3 in one run and
paragraph 5 in another, side b attributed only paragraph 3) are not reported as
differing, since that overlap already contains agreement — the record only exists for the
claim that is purely between models, with within-model variation left to
stability.*. Both sides' complete observed sets are always carried on the
record, so a reader wanting a weaker "disagreed at least once" relation can derive it without
re-reading cells/**.
locator: no byte span, ever
locator.paragraph is a crate::paragraph::split index — the minimum
paragraph either side observed for that key — never a byte offset. Byte spans are not
persisted anywhere in this pipeline; chunk_index/chunk_sha256 are
derived from the paragraph index via manifest.json's own chunk dictionary at
diff time, and are null whenever no chunk covers it (an older manifest, or
--no-passage having stripped every locator upstream). text stays
null unless --with-text was given; when it is,
compare re-reads the corpus file manifest.json names, verifies it
against the pinned document_sha256, and slices out the paragraph — a loud
refusal (naming the file and the mismatched hash, exit code 1, nothing
written) if the file is unreadable or the hash no longer matches, since text the models never
saw must never be embedded silently. Embedded text is capped at 4096 bytes (floored to a
character boundary), with text_truncated: true marking the cut.
taguru benchmark search
taguru benchmark search --eval eval.jsonl --url http://localhost:8248 \
results/
--eval FILE eval.jsonl (the shared dataset below)
--url URL server to import into and search; default resolves the
same way `taguru health` does (TAGURU_ADDR, or
--config/TAGURU_CONFIG)
--config FILE load before resolving --url
--run N which extract run to import from, 1-based (default 1)
--context-prefix NAME corpus context names are NAME::MODEL_ID (default: the
extraction context manifest.json recorded)
--skip-import search existing PREFIX::MODEL_ID contexts instead of
importing fresh ones
RESULTS_DIR a directory `taguru benchmark extract` wrote
Extraction-model choice does not only change what a graph contains — it changes what a
search finds. taguru benchmark search (issue #260, ADR 0003 §11) builds one
context per model from a finished results directory's already-written batch files, runs one
shared question set against every one of them over the exact retrieval path a real client
uses (POST /contexts/{name}/sources/search), and writes
results/retrieval.json: per-case, per-model hit counts, lane evidence, and
model-pair hit-set overlap. Like differences.jsonl,
it is gold-data-free and judgment-free by construction — with no expectations supplied
it reports only differences between corpora, never which model's results are better. Unlike
taguru benchmark compare, it is not a pure function of
the results directory: it imports into and queries a running server, so a reachable
server is required, and re-running is idempotent (a per-source create-or-replace) rather than
a byte-identical rewrite.
Exit codes: 0 written · 1 the server is unreachable, or every
model's corpus failed to build or search · 2 usage error, an unreadable
eval.jsonl, a --run the manifest never recorded, or a corpus
context name over 64 bytes.
Per-model corpora
For the requested --run, every model whose cell recorded outcome:
"complete" has its cells/<model_id>/run<NN>/*.jsonl batch
files (diagnostics.jsonl excluded) imported into a context named
PREFIX::MODEL_ID. Each batch's header line is rewritten before import — its
context replaced, and its create.description stamped with an
ownership marker naming this run's run_id, the model, and --run's
own index — every other line rides through byte-for-byte, source included.
run_index is part of the marker, not just run_id, because a results
directory's run_id names the whole taguru benchmark extract
invocation, the same across every --run N within it — without
run_index in the marker, running --run 1 then --run 2
against the same directory would pass the ownership check unchanged and silently mix both
runs' documents into one corpus (import never removes a source absent from the current
import's own batch set). Re-running the exact same --run against the same
results directory is safe: import is a per-(context, source) create-or-replace,
so a second run neither duplicates associations nor loses them. Before importing into an
existing context, its current description is checked against the marker this run would
stamp; a mismatch — including a different --run — refuses that one model
(corpus.<model_id>.outcome: "rejected") rather than overwriting a context
this run did not create or merging a different run into it — pick a different
--context-prefix if that happens. --skip-import searches whatever
already lives under
PREFIX::MODEL_ID instead, useful for re-running only the search-and-measure half
after a corpus was built once.
eval.jsonl: the dataset shared with the evaluate quality gate
{"taguru_eval":1,"name":"sake retrieval cases","default_target":{"context":"sake"}}
{"case_id":"brand-origin-001","query":"青嶺はどこの蔵の酒か","cues":["青嶺"],
"expected_sources":[{"source":"corpus/brewery.md","paragraphs":[0],"relevance":3}],
"expected_concepts":["青嶺酒造"],"options":{"limit":10}}
One taguru_eval header (equality-checked — this file is hand-authored, so a
version this build was not built for is refused rather than silently reinterpreted, the same
posture batch import's taguru_batch takes), then one
case per line. Every case field this command does not know is still declared and accepted —
never rejected as a typo — since eval.jsonl is the one artifact this reference
shares with taguru evaluate's own quality gate over
an already-populated corpus: expected_labels,
expected_associations, expected_citations, and
options.floor/sources/since are #215-only and ride
through untouched, earning one warning per run (never once per case) if seen —
evaluate.html documents what those fields mean and how
evaluate itself interprets and validates them. The fields this
command reads: case_id (a stable, unique join key), query (sent to
sources/search verbatim), expected_sources[] (source
matched against the results directory's own document dictionary — an exact path, then a
document_id, then a unique path suffix, in that order; empty
paragraphs means any paragraph of that source counts; relevance 0
excludes the entry), expected_concepts[], and options.limit (the
per-case search limit; omitted means 10). cues and
target are echoed into retrieval.json's own per-case block for a
reader's reference but not otherwise used — this command always overrides the search target
with its own per-model corpus, one context per model, so a shared dataset naming no target at
all is the normal case.
retrieval.json
{
"taguru_benchmark_retrieval": 1,
"run_id": "...", "generated_at": "2026-07-26T10:05:12Z",
"matching": { "module": "benchmark::identity", "case_fold": true,
"unicode_normalization": "NFKC", "alias_expansion": "batch-local", "weight_tolerance": 0.0 },
"inputs": { "results_dir": "results/", "eval": { "path": "eval.jsonl",
"name": "sake retrieval cases", "cases": 12 }, "url": "http://localhost:8248",
"run_index": 1, "default_limit": 10 },
"definitions": { "hits.count": { "unit": "count", "statistic": "distribution", "...": "..." }, ... },
"warnings": [],
"corpus": { "qwen25-7b-q4": { "context": "sake::qwen25-7b-q4", "outcome": "built",
"documents_imported": 5, "documents_failed": 0,
"passage_vectors": { "model": null, "rows": 0 } } },
"cases": [ { "case_id": "brand-origin-001", "query": "青嶺はどこの蔵の酒か", "cues": ["青嶺"],
"limit": 10, "has_expectations": true,
"models": { "qwen25-7b-q4": { "outcome": "searched", "hit_count": 3, "empty": false,
"distinct_sources": 2, "lanes": { "bm25_only": 1, "vector_only": 0, "both": 2,
"neither": 0, "unknown": 0 },
"plan": { "bm25": { "ran": true }, "vector": { "ran": false,
"reason": "no embedding provider is configured" } },
"hits": [ { "rank": 1, "source": "corpus/brewery.md", "paragraph": 0 } ],
"recall": { "recall_at_k": 1.0, "mrr": 1.0, "expected_total": 2, "matched": 2 } } },
"pairs": { "hosted-120b__qwen25-7b-q4": { "outcome": "compared", "jaccard": 0.33,
"shared_hits": 1, "mean_rank_difference": 1.0 } } } ],
"models": { "qwen25-7b-q4": { "hits.count": { "n": 12, "...": "..." }, "...": "..." } },
"pairs": { "hosted-120b__qwen25-7b-q4": { "overlap.jaccard": { "n": 12, "...": "..." }, "...": "..." } }
}
taguru_benchmark_retrieval accepts any version from 1 through the
current one, like manifest.json — this is taguru's own written-and-re-readable
artifact, not a hand-authored input. No CSV is written; a static
report.html (issue #261, optional) is where visualization belongs.
recall@k and MRR: the one thing shared with the evaluate quality gate
When a case carries expected_sources or expected_concepts, both are
folded into one recall@k/MRR computation, computed from the very
sources/search hits already fetched for that case — never a second retrieval
call. A hit satisfies an expected_sources entry when its source
matches (resolved against the document dictionary as above) and, when paragraphs
is non-empty, its paragraph is among them. A hit satisfies an
expected_concepts entry when a case-folded, NFKC-normalized copy of the concept
string is a substring of a case-folded, NFKC-normalized copy of the hit's own text.
recall_at_k is the fraction of a case's expected entries (source and concept,
together) satisfied by at least one hit; mrr is 1 / rank of the
first hit that satisfies any expected entry, 0 if none does. Cases with no
expectations at all carry no recall block — this reference reports only
differences for them, exactly as ADR 0003 §11 frames the gold-data-free tier.
taguru evaluate builds on exactly this pair
with graded-relevance nDCG and citation metrics on top, and — unlike this reference — turns
the result into a pass/fail gate.
Why not POST /contexts/{name}/recall
expected_concepts is checked against hit text, not driven through the graph's own
recall endpoint, on purpose: Context::recall(cue) requires
cue to equal a stored concept or label id exactly — it performs no fuzzy
or semantic resolution — so a hand-written eval.jsonl's natural-language
cues[] would silently miss almost everything. ADR 0003 §11 names exactly one
retrieval endpoint for this command, sources/search, and this keeps to it: one
search call per case per model, never two.
Lane evidence and the embeddings/refresh gap
plan and per-hit lane accounting are read straight off sources/search's
own response (the same plan/lanes shape every client sees —
documented in full, with a worked example, under POST /contexts/{name}/sources/search
in the LLM-facing protocol manual served at GET /protocol) — this command never calls
POST /contexts/{name}/embeddings/refresh, because that endpoint only re-embeds
concept and label glosses (what resolve's semantic fallback reads), never
the passage vectors sources/search's vector lane actually reads. Passage
embeddings refresh only from the server's own background flush tick, gated on
TAGURU_EMBED_PASSAGES, and have no on-demand HTTP endpoint at all — so whether the
vector lane runs for a given corpus is outside this command's control. corpus.*.passage_vectors
is a best-effort, read-only GET /contexts/{name}/embeddings snapshot taken once
per corpus, offered purely as evidence for why a case's plan.vector.reason reads
the way it does. A response missing plan/lane fields entirely (an older server)
degrades to lane fields recorded null — every non-lane comparison for that case
is still written.
Metric catalog
| Metric family | Scope | What it measures |
|---|---|---|
hits.count | model | Passage hits sources/search returned for a case, truncated to that case's limit. |
hits.empty_rate | model | Share of successfully searched cases whose search returned zero hits. |
hits.distinct_sources | model | Distinct source documents among a case's hits — a coarse diversity signal. |
lanes.bm25_only, .vector_only, .both, .neither | model | Hits a case's search found via the lexical lane only, the semantic lane only, both agreeing, or (readable but) neither. |
lanes.unknown | model | Hits whose per-hit lane evidence could not be read at all (an older or otherwise non-conforming server) — kept apart from the four above so a legacy response's hits are never misread as evidencing zero lane activity. |
cases.error_rate | model | Share of cases whose search request failed outright — a network/HTTP failure, or the model's corpus was never available. |
recall.recall_at_k, .mrr | model | As defined above, over cases carrying at least one expectation. |
overlap.jaccard | pair | Jaccard similarity of two models' hit (source, paragraph) sets for the same case — None, not a vacuous 1.0, when both sides returned zero hits. |
overlap.shared_hits | pair | Locators both models' hits agreed on, for one case. |
overlap.mean_rank_difference | pair | Mean absolute difference in hit position (1-based) among locators both models found. |
pairs.unavailable_rate | pair | Share of cases where at least one side of the pair could not be searched. |
No verdict vocabulary
A model-pair's key is its two model ids joined in sorted order ("m1__m2"), never
baseline/candidate — the same posture
measurements.json/differences.jsonl hold. winner,
best, recommended, overall, and delta_vs_*
never appear anywhere in this artifact. hits[].rank is the one deliberate
exception to the wider rank/score key-fragment ban
measurements.json carries: it names a hit's own position in one model's result
list — the same ordinary IR sense LaneEvidence.rank already uses on the wire — not
a cross-model ranking, so it is not treated as a banned word here.
Taguru