The batch contract — taguru import and POST /import

The CLI shape

taguru import applies batch files straight to the data directory — without a running server, and never beside one: the directory lock makes them mutually exclusive (an advisory flock — dependable on local disks; a network filesystem may grant it to both sides). Point it at a running server instead with --url.

taguru import batch/                  # every *.jsonl under batch/, in name order
taguru import --dry-run batch/        # validate and report; touches nothing
taguru import --config prod.env a.jsonl b.jsonl
taguru import --url http://127.0.0.1:8248 batch/   # send to a running server instead
taguru import --json batch/           # {dry_run, batches: [...], schemas: [...], groups: [...]} instead of text

Exit codes: 0 everything applied · 1 something was refused or failed (details on stderr) · 2 usage error.

--no-embed is offline-only — combined with --url it is a usage error, since the server's own TAGURU_EMBED_URL configuration decides once the request lands there.

--json answers the same {batches, schemas, groups} fields POST /import's own result object carries — one ImportOutcome per batch under batches, one SchemaImportOutcome per taguru_schema record under schemas, one GroupImportOutcome per taguru_group record under groups (both omitted entirely when the stream carried none) — so a script parses one schema whether it called the CLI or the HTTP endpoint directly. The two outer shapes differ: the CLI prints {"dry_run": ..., "batches": [...], ...} directly, while the HTTP endpoint wraps the identical fields inside its own response envelope, {"status": "ok", "result": {"batches": [...], ...}, "time": ...} (or {"status": "error", "error": "...", ...} on refusal — every JSON endpoint answers this same envelope). With --url, every field is exact: the server always answers from the same code path a real apply runs (preview_batch under --dry-run). Offline, a real (non-dry-run) run is exact the same way; offline --dry-run is the one exception — it never boots the registry (so it needs no directory lock to validate files), and without a boot it cannot know created/retracted or any *_dropped count the way a real apply can. Those fields read 0/false rather than a guess, and schemas/ groups are always absent for the same reason the HTTP endpoint's own ?dry_run=true omits them: neither installing a schema nor restoring a group has a read-only twin to preview through.

Every --json exit path prints exactly one JSON document, failures included — never silent stdout on a non-zero exit. Validation refusing every file, the registry refusing to boot, and a remote transport error or a server-refused chunk each add a top-level error string beside batches/schemas/ groups, unchanged from the success case's own rule: batches is always an array ([] when nothing landed before the failure), schemas/groups are each omitted whenever they would be empty — a failure that landed zero batches, zero schemas, and zero groups still answers {"dry_run": ..., "error": "...", "batches": []}, no schemas or groups key at all. A local batch refused mid-run (offline only — a remote refusal fails its whole chunk, never one batch within it) is named under a failed_batches array instead of appearing in batches, since there is no ImportOutcome to build for a batch that was never applied; the human-readable stderr line for the same failure is unchanged.

One file = one source's complete truth

Applying a file means: retract that source, then apply the file. This one rule buys every operational property below.

• Idempotent — importing the same file twice lands in the same state. Weights never double-count.
• Revisable — a corrected file for the same source replaces the old facts wholesale. The same operation as the live diff-sync an agent performs (retract_source → re-ingest).
• Retryable — a file that failed midway (capacity, disk) is fixed and re-imported. The retraction is what makes the retry exact.

This is also why an association line cannot carry its own source field (it is refused): a source the header does not declare would survive the retraction and double itself on every re-import. The header's source is stamped onto every association in the file.

A file that is only a header, with no operation lines, is a pure retraction — "this source no longer asserts anything."

One sharp edge. Because of the retract-then-apply order, if the apply stage fails persistently (capacity is the realistic case), that source stays empty until a re-import succeeds (old facts retracted, new facts refused). Also, retraction releases a source's attributions but does not un-register vocabulary: once minted, a concept or label keeps its id for the lifetime of the context. A pipeline that keeps re-importing revision files whose names keep rotating grows the context monotonically. Watch the counts with taguru inspect and the headroom with taguru estimate; accumulated dead weight is reclaimed by taguru compact (offline) or POST /contexts/{name}/compact (live, admin) — live facts, counts, and paragraph locators are all preserved; only retracted records and arena debris disappear.

The file format

JSON Lines, UTF-8. Blank lines are ignored. The first non-blank line is the header; every following line is one operation. (A stream may also carry taguru_schema and taguru_group records — see below.)

{"taguru_batch": 1, "context": "sake", "source": "docs/aomine.md", "create": {"description": "酒蔵の知識"}}
{"passage": "青嶺酒造は1907年創業。杜氏は高瀬。", "date": 1700000000, "tags": ["会社概要"]}
{"paragraph": 0, "section": "沿革"}
{"paragraph": 0, "locator": {"kind": "page", "value": "12"}}
{"paragraph": 0, "question": "青嶺酒造の酒造りの責任者は誰?"}
{"subject": "青嶺酒造", "label": "創業年", "object": "1907年", "weight": 1.0}
{"subject": "青嶺酒造", "label": "杜氏", "object": "高瀬", "weight": 2.0}
{"alias": "Aomine Brewery", "canonical": "青嶺酒造", "kind": "concept"}
{"alias": "設立年", "canonical": "創業年", "kind": "label"}

Header

taguru_batch must be 1 (the only version this build reads). context and source are required. create is optional and carries the same fields as PUT /contexts/{name} (description, pinned, dice_floor, semantic_floor) — used only when the context does not exist yet. Importing into an absent context without a create block is refused.

Association lines

subject / label / object / weight — exactly the fields of the associations endpoint minus source (for the reason above).

Alias lines

alias / canonical / kind ("concept" or "label"). Aliases are context-level, not source-scoped: re-import re-registers (an unchanged pair is a no-op), and retraction does not remove aliases. The canonical may be a name first introduced by this file's associations — associations apply first. Re-pointing an existing alias at a different canonical is a conflict and fails the file — import must never silently rewire vocabulary. The fix is deliberately two-step: retract the old registration first (DELETE /contexts/{name}/aliases with the exact spelling, or MCP's remove_aliases), then re-import — the retraction makes the retry exact. A batch file itself cannot delete an alias. Also deliberate.

Passage line

The source's original text. At most one per file. Stored behind the same source id and served by sources/lookup and sources/search. The same line optionally carries the source's metadata: date (the document's own time, epoch seconds — what a search's since/until window prefers over the stamp below), tags (a string array, capped like other batch fields), and stored_at (epoch seconds). A freshly authored file should leave stored_at out — the server stamps it itself as the write lands; taguru export fills it back in on the way out purely so a restore does not re-date the corpus. All three are optional and independent of each other.

Question lines (doc2query)

paragraph / question: a retrieval question attached to one paragraph of this file's passage. paragraph is a 0-based position in the server's blank-line split (a question line in a file with no passage line is refused). One question per line; several may target the same paragraph (up to the cap). A question's terms index into its paragraph's BM25 postings on every server, and a server with TAGURU_EMBED_PASSAGES also embeds it next to the paragraph — either way, question-shaped searches land on answer-shaped text. A question naming a paragraph that does not exist in the passage's split is dropped at store time and counted in the response (questions_dropped) — the usual cause is a producer's split drifting from the server's. taguru extract --questions N uses the server's own splitter, so it cannot drift.

Section lines

paragraph / section: a heading marking where a named section starts within this file's passage (0-based, same split convention as questions; refused without a passage line). A section implicitly extends from its paragraph to the start of the next section or the end of the text — there is no "end" line. Sections naming paragraphs absent from the split are dropped at store time and counted (sections_dropped; same usual cause as questions). An association whose attribution carries a matching paragraph locator reports the section on that attribution: recall, query, explore, activate, and unreachable_from all resolve attributions[].section, or null when the locator falls in no section. POST /contexts/{name}/citations performs the same resolution for its one paragraph. Passage-search hits do not read this field yet and keep omitting it regardless of what is stored — a separate follow-up.

Locator lines

paragraph / locator: {kind, value}: a typed citation locator (ADR 0007 §7) — a page, slide, sheet, table, or other position, independent of section and stored alongside it (0-based, same split convention as questions/sections; refused without a passage line). kind is an open string; the standard ingest connectors populate it as "page" (PdfConnector), "fragment" (HtmlConnector), "table" (DocxConnector), and "slide"/"speaker_notes" (PptxConnector) — see the full mapping. value is free text since its natural representation varies by kind ("12" for a page, "installation" for an HTML fragment id). Unlike a section, a locator does not extend to the next paragraph — it names only the exact paragraph it was recorded for. Locators naming paragraphs absent from the split are dropped at store time and counted (locators_dropped; same usual cause as questions/sections). The same resolution reaches attributions[].locator on recall, query, explore, activate, unreachable_from, and POST /contexts/{name}/citations's locator, exactly like section.

Schema records

This is the record's on-the-wire shape only. For what the document means, the three modes, the reserved schema:type label, and the write-time enforcement it turns on, see The context schema.

{"taguru_schema": 1, "context": "…", "mode": "off"|"warn"|"strict", "closed_labels": …, "types": {…}, "relations": {…}} — one line states one context's whole schema document (ADR 0009 §13), the same shape PUT /contexts/{name}/schema installs. Unlike a group record, every field is required — there is no default for an absent one, matching the document's own at-rest shape: a missing field is a parse refusal, never a silently-defaulted one. A schema record stands alone between batches (it closes the batch before it, the same as a group record — operation lines after one need a fresh header), and taguru export emits one as the first line of a context's own stream whenever that context has a schema installed and its mode is not "off" — a schema left in off while its types are drafted is not carried across export/restore.

Schema records install after every batch of the run, before any group record — so a schema can name a context a batch of the same stream just created, and a group record can in turn rely on that schema having already landed. Each record is independent (one record, one context): unlike group records, which validate as one whole set before any of them applies, the first schema record that fails to install refuses the request right there — every batch before it stays durable, and later schema records or group records in the same stream are simply never reached. Restating one context's schema twice in a run is refused — one record owns one context's schema. The response's SchemaImportOutcome carries no outcome verb (unlike a group's "created"/"replaced"/ "unchanged"): installing a schema cannot itself tell an install apart from a no-op re-install of the identical document, and a guessed verb would be worse than none.

--url carries one more preflight for this record kind: before a byte ships, the CLI reads the peer's GET /version schema_formats and refuses up front — naming both sides — when the peer cannot read this build's schema format. A schema-free stream never triggers the check and behaves exactly as it did before this preflight existed.

Group records

{"taguru_group": 1, "name": "…", "description": "…", "contexts": […], "groups": […]} — one line states one group's complete truth, the way a batch states a source's: applying it is a create-or-replace of the whole record, never a delta, so re-importing stays idempotent. Only name is required; absent fields read as empty. A group record stands alone between batches (it closes the batch before it — operation lines after one need a fresh header), and taguru export writes each group as its own {group}.group.jsonl beside the context streams.

Groups apply after every batch and every schema record of the run — one CLI invocation, one POST /import body — wherever the records sat, so a group and the member contexts it names can travel together in any order. Referential integrity holds at that point exactly as over the API: every member context must exist, every child group must exist or ride the same run, the nesting must stay at most 3 groups tall and acyclic, and a violation refuses all group records (the batches and schemas are already durably applied; fixing the stream and re-importing it whole is exact). Restating one group twice in a run is refused — one record owns one group's truth.

Unknown fields and unrecognizable line shapes are refused, with the line number.

Caps (the same ones as the API, enforced per line)

There is no cap on operations per file. Apply is internally chunked at the API batch size (10,000) and long runs flush periodically, so no context's WAL ever approaches TAGURU_WAL_MAX_BYTES.

Validate, then apply

• Every file parses, and no two files claim the same (context, source) — only once that is established does anything apply. One malformed line refuses the entire run with its line number, and nothing is written (--dry-run stops there by design).
• Failures only discoverable at the apply stage (capacity, disk) are reported per file, and the remaining files continue: files are structurally independent (one file, one source), and a partially applied file heals on re-import.
• A batch that stops between its four durable steps (retract → passage → associations → aliases) is detected, not just repairable: a per-source batch-open marker is written before the first step and removed only after the last, so the server's next boot and taguru inspect name any source whose import never completed. An alias step predicted to conflict is refused before this marker ever opens — canonical/alias resolution is simulated read-only against the batch's own fresh names and the context's current state — so a content-driven refusal leaves nothing to detect; the marker's presence now narrows to a capacity or disk fault reaching mid-batch. Either documented repair clears it — re-import the file, or retract the source. Cross-store atomicity is deliberately not attempted; retract-then-apply already makes the repair exact.
• Writes go through the same registry the server uses — staged on the WAL and fsynced, budgets enforced, one flush at the end. So taguru inspect and the next boot see exactly the state a live ingest would have produced. Imports count as writes in each context's usage stats.
• If the target context has an installed schema, every batch is also judged against it: strict refuses the whole batch alongside the checks above (before anything mutates), warn applies it and reports the violations in the response's issues. ?dry_run=true runs the identical schema check, so a preview can never disagree with the real apply.

The operational procedure for a marker left by an interrupted bulk load — diagnosing it, choosing a repair, and folding that into a resumable runner's failure handling — is in long-running ingestion.

Validating everything first has a cost: every parsed file stays in memory until apply, so the run's footprint is proportional to the total size of the file set (not the largest one). Stream a million-document migration as a few invocations (slice the directory) — import is idempotent and every file carries its own source, so splitting costs nothing.

Embeddings

If TAGURU_EMBED_URL/TAGURU_EMBED_MODEL are set (also via --config), every touched context re-embeds its changed glosses at the end of the run — the same idempotent refresh as the server's TAGURU_EMBED_AUTO. --no-embed skips it. A refresh failure rolls back nothing: the graph is imported and durable, and the run exits 1 naming the recovery step (POST /contexts/{name}/embeddings/refresh on a running server).

The same contract over HTTP: POST /import

The request body is one batch file, and one request states one source's complete truth. A live system bulk-loads with no downtime window.

# --data-binary, not -d: curl's -d strips newlines, and newlines are the format
curl -X POST localhost:8248/import \
  -H 'Authorization: Bearer <key>' \
  --data-binary @docs-aomine.jsonl
# → {"result": {"context": "sake", "source": "docs/aomine.md",
#    "created": false, "retracted": 12, "associations": 14,
#    "aliases": 2, "passage_stored": true}, "status": "ok", …}

• Validation first (a malformed line is a 400 with its line number, and nothing applies), then retract-then-apply — re-POSTing the same batch replaces the source exactly, just like re-running the CLI. The response carries the same numbers the CLI prints.
• It obeys everything every endpoint obeys: Bearer auth, the body cap (TAGURU_MAX_BODY_BYTES, default 8 MiB — larger batches go through the offline CLI or a raised cap), the request timeout, per-key rate limits.
• Statuses follow the API's dialect: 400 malformed batch (with line number) · 404 absent context and no create block · 409/507 partial write (conflict / capacity; the message names what landed, and the retraction makes the corrected retry exact) · 500 a write the WAL or disk refused.
• A mid-stream refusal also says what landed machine-readably: the error body carries integrity ("nothing_written", or "durable_prefix" with durable_batches naming exactly how many batches before the refused one are durable — a dry run is always nothing_written) and, for correctable streams, retryable_after_correction: true. An automated importer resumes by skipping the durable prefix, or re-POSTs the corrected stream whole — retract-then-apply makes either exact. Timeouts (408, budget spent partway) carry integrity and durable_batches too but omit retryable_after_correction — there is nothing to correct; resume (or re-POST as-is) with more budget.
• Embeddings are not refreshed within the request. With TAGURU_EMBED_AUTO on, they ride the next flush, same as live writes.

Two batches for different sources may be POSTed concurrently — each write inside a batch is serialized on the context. Never race two imports of the same source: one side's apply can interleave with the other's retract, and the union of both batches can survive. One source, one writer at a time.

Choosing the entrance: no server yet, or a huge initial load → taguru import (no per-request caps, one flush at the end). A live system that must keep answering → POST /import directly, or taguru import --url below for a whole file set at once.

import --url: chunked over the wire

taguru import --url URL FILE|DIR... drives POST /import for a whole file set the same way taguru import drives the local write path — without the operator hand-splitting anything to fit under the body cap.

taguru import --url "$TAGURU_URL" backups/
taguru import --url "$TAGURU_URL" --dry-run backups/   # preview every chunk first

• The input is split on batch boundaries only, never mid-batch: a batch runs from its taguru_batch header line to the next taguru_batch/ taguru_schema/taguru_group line or EOF, the exact rule taguru router's own cross-shard import splitting already uses. Whole batches are packed into each request up to a byte budget starting at 4 MiB (half the 8 MiB default TAGURU_MAX_BODY_BYTES, leaving headroom for a server configured lower).
• A single batch that alone exceeds this fixed, client-side budget is a hard error naming the source and the one real fix: reduce what that source's batch carries (split the source upstream of import). Raising the server's TAGURU_MAX_BODY_BYTES alone does not help — this budget check happens before the server is ever asked. Splitting a batch's own record set client-side is not an option either — it would reimplement the retract-then-apply contract's atomicity boundary outside the server that owns it.
• A 413 on a chunk still oversized only because the cap is configured lower than assumed halves that chunk (never crossing a batch boundary) and resends, halving again on every further 413 until the chunk lands or hits a single batch — safe to automate since the server refuses a 413 before applying anything.
• taguru_schema records ride after every batch chunk, and taguru_group records after every schema chunk in turn — the same order the local path already applies them in, so a schema naming a context an earlier chunk's batch creates, or a group naming a context an earlier chunk's schema installs onto, always arrives after it.
• A stream that actually carries a taguru_schema record gets one more preflight first: the CLI reads the server's GET /version schema_formats and refuses up front, naming both sides, if the server cannot read this build's schema format — nothing is sent. A schema-free stream never triggers this check.
• A lost connection reports which chunk landed and points at --dry-run to confirm what remains before resuming — nothing past that point is retried automatically. Import's retract-then-apply contract makes any resend exact, so this is a "where," never a "was it safe," question.
• Auth rides TAGURU_API_TOKEN (or the first name:token entry of TAGURU_API_TOKENS), the same variables the server itself reads, and the admin role POST /import requires.

Batch streams and taguru export (backup / restore)

One file (one request body) may carry one batch or several concatenated ones — each taguru_batch header line closes the previous batch and opens the next. Per-batch validation, retract-then-apply, and (context, source) uniqueness are identical to the single-file case.

That stream is what taguru export --out DIR [CONTEXT...] writes (offline, under the directory lock), and what a running server serves at GET /contexts/{name}/export (the body is the JSON Lines stream itself). One context = one stream: one batch per source id, in order, with a create block (description, pinning, floors) on the first batch and the alias table riding the last. A context whose schema is installed and not in off mode carries one more line — a taguru_schema record, first in the stream (ADR 0009 §13) — so its constraints travel with the data rather than being a separate, easy-to-forget backup step; a schema left in off while its types are drafted is not exported. Restore is a normal import — per-source replacement makes it idempotent; to match a snapshot exactly, delete the context first, then stream it in.

taguru export --url URL --out DIR [CONTEXT...] automates the same walk against a running server instead of a local data directory: it enumerates GET /contexts — and, without CONTEXT arguments (a full export), GET /groups too, both keyset-paged — fetching each item from its own export endpoint and writing the identical files a local export would. Auth rides TAGURU_API_TOKEN (or the first name:token entry of TAGURU_API_TOKENS), the same variables the server reads; a URL carrying user:password@ is refused. This is not a point-in-time snapshot across contexts — each context's own stream is internally consistent, but contexts are fetched one request at a time, so two contexts fetched seconds apart may not reflect the same instant. An operator who needs a true point-in-time snapshot across the whole server already has one: the replication bucket (TAGURU_REPLICATE_URL) plus taguru restore.

• Fidelity: a fact the same source asserted n times reproduces as n association lines (corroboration count survives the round trip). Paragraph locators ride the first assertion line (reproducing first-write-wins).
• Sourceless weight (entered via the API with source omitted) rides a batch under the reserved source export:unsourced — the numbers are preserved and the attribution tells the story. If a real source collides with this reserved id, export refuses rather than silently mixing.
• Fully retracted edges (weight 0) are not emitted: an export → import round trip sheds the dead records an append-only image accumulates — doubling as offline compaction.
• An alias whose canonical has no live associations is dropped and the count reported (under the import contract, canonicals are minted by association lines).
• POST /import always answers {"status": "ok", "result": {"batches": [per-batch results…]}, "time": ...} (a single batch is the same shape with one element — one shape to parse), result gaining schemas/groups whenever the stream carried either. If the stream is refused midway, the earlier batches (and any schema record that already installed) have landed — each batch is retract-then-apply and each schema install is independent, so re-POSTing the corrected stream converges exactly.
• Rendering is deterministic: the same state writes the same bytes, so backups diff against each other.

# evacuate one context from a live server and restore it elsewhere
curl -H 'Authorization: Bearer <key>' \
  localhost:8248/contexts/sake/export > sake.jsonl
curl -X POST other-host:8248/import \
  -H 'Authorization: Bearer <key>' --data-binary @sake.jsonl
# offline:  taguru export --out backups/ ; taguru import backups/
# whole server, live — the loop above, automated:
taguru export --url "$TAGURU_URL" --out backups/

Producing batch files

Taguru's server does not extract facts from prose — that is the reading LLM's job, and the server never holds model credentials. A producer reads the documents, has a model emit associations under the /protocol discipline, and writes one batch file per document with the document id as the source.

One such producer ships in the box: taguru extract — an offline subcommand that does exactly the above against any OpenAI-compatible chat endpoint, holding the TAGURU_EXTRACT_* credentials in its own environment only. Custom pipelines remain first-class: the Converse loop in the Bedrock guide and the tool vocabulary of /protocol are the two halves of such a producer, and a batch file is merely its offline serialization.