Modeling — mapping real documents onto contexts and groups

Five words, five responsibilities

Which unit owns which job. Everything below follows from this table.

Chunk is not a server concept. Taguru never reads or stores a "chunk" — it appears only inside taguru extract, the offline producer that turns documents into batch files, as a bound on how much text one model call can see. Once ingest happens, the unit that exists on the server is the passage (a source's full text) split into paragraphs. A chunk boundary that survives into the graph as a context boundary is an accident of the pipeline, not a decision about meaning.

Where the context boundary falls

Three questions, in order, when a context's edge is not obvious.

A worked example: one paper

A paper's own structure — sections, chapters, front matter — maps onto contexts and groups directly, once each unit's job is named.

group: paper/bert                      # the paper — a group
├── context: section/bert/abstract     # front matter: no chapter owns it
├── group: chapter/bert/2              # a chapter — a group of its sections
│   ├── context: section/bert/2        # the chapter's own lead-in text, if any
│   ├── context: section/bert/2.1
│   └── context: section/bert/2.2
└── group: chapter/bert/3
    └── …

Why the section is the context

A section is a coherent frame: its vocabulary stays stable within it, which is exactly the one-spelling-one-referent contract a context exists to hold. A chapter carries no text of its own — it only bundles the sections beneath it — so it is a group, not a context. The paper is the same shape one level up: a group whose members are chapter groups, plus whatever front matter (the abstract) never belonged to a chapter in the first place. Nesting here is two groups deep (paper → chapter), one below the 3-group ceiling, leaving room for a collection above the paper if one is ever needed.

Front matter, and sections too long to ingest at once

The abstract belongs to the paper but to no chapter, so it sits as a context directly under the paper/bert group, alongside the chapter groups — a group's members are contexts and child groups at once, not a choice between them.

A section too long for one ingest pass does not become two contexts. It stays one context — the frame of meaning has not changed — and gains a second source: bert/2.2/part-1, bert/2.2/part-2, both inside section/bert/2.2. Each part retracts and re-imports on its own; the section's vocabulary and aliases stay shared across both, which is the entire reason splitting by source rather than by context is the correct move here.

Building it with one import stream

A run of taguru import (or one POST /import body) can carry several batches and any number of taguru_group records together — group records apply after every batch in the run, so a group and the contexts it names can travel in any order. One file for the two parts of §2.1, closing with the chapter and paper groups:

{"taguru_batch": 1, "context": "section/bert/2.1", "source": "bert/2.1/part-1", "create": {"description": "BERT §2.1 — Pre-training BERT"}}
{"passage": "We pre-train BERT using two unsupervised tasks…"}
{"subject": "BERT", "label": "pre-training task", "object": "masked LM", "weight": 1.0}
{"taguru_batch": 1, "context": "section/bert/2.1", "source": "bert/2.1/part-2"}
{"passage": "…the second task jointly pre-trains text-pair representations."}
{"subject": "BERT", "label": "pre-training task", "object": "next sentence prediction", "weight": 1.0}
{"taguru_group": 1, "name": "chapter/bert/2", "contexts": ["section/bert/2", "section/bert/2.1", "section/bert/2.2"]}
{"taguru_group": 1, "name": "paper/bert", "contexts": ["section/bert/abstract"], "groups": ["chapter/bert/2", "chapter/bert/3"]}

The second batch's create block is left out on purpose — the context already exists from the first batch, and importing into an absent context without a create block is refused, so this is also how the parser confirms both parts land in the same context rather than silently minting two. Full field-by-field contract: The batch contract.

Searching across papers names groups, not a super-context

A question that spans several papers names the papers' groups at search time — it never needs a context that contains them all.

POST /recall, POST /query, and POST /sources/search take contexts and groups lists together and run one search across every context reached, nested children included, every match tagged with the context it came from:

curl -X POST localhost:8248/recall -H 'Content-Type: application/json' \
  -d '{"groups": ["paper/bert", "paper/gpt"], "cue": "fine-tuning"}'

The response's plan names exactly which contexts the two groups expanded to, so "which paper answered" is never a guess. Deep dives — activate, explore, paths, resolve, and the composite retrieve tool — stay per-context by design (see Concepts): search across with a cross call, then pull the thread inside whichever context answered. Reaching for retrieve against one context after deliberately splitting a corpus into several undoes the split; a LangChain retriever crossing the same set is TaguruRetriever(groups=["paper/bert", "paper/gpt"]), not a loop of per-context retrievers. Run end to end against real PDFs — extraction, this cross-group retriever, and a local answer model — in the local RAG walkthrough.

None of this needs a group that exists only to be searched. An ad-hoc cross-paper question names the papers' groups directly in the request; a group is worth creating only for a collection reused across many searches (a reading list, a chapter's sections). A root group built solely so "search everything" has one name to type adds a namespace with no boundary behind it.

The slash in a name is typography, not structure

Membership is the only hierarchy the server acts on.

paper/bert and section/bert/2.1 are opaque strings, capped at 64 bytes, from a namespace the server never parses. Contexts and groups are separate namespaces from each other, too. The / reads as a path to a person; to the server it is one more byte. Writing paper/chapter/section into a source id is the same mistake one level down — a source is a provenance id scoped to one context, not an address, and the server never decomposes it into a location.

Membership is the only hierarchy. A name that looks like a path is a naming convention for humans reading a list. A group whose contexts or groups field names something is a fact the server enforces — referential integrity, nesting depth, cycle checks all apply to it. Only the second kind of statement is structure.

Bodies (a recall request, an import batch's context field) carry a slash-bearing name as-is. Only when the name itself sits in the URL path — PUT /contexts/{name}, PUT /groups/{name} — does the slash need percent-encoding (paper%2Fbert), the same as any other path segment; otherwise it would split the route in two.

The failure mode: a context grown into a corpus

What actually breaks when several papers share one context, in the order it breaks.

• Alias collisions. Aliases live in the context's own vocabulary (Concepts), and one spelling can never shadow an existing name or alias. Two papers that each abbreviate a different term the same way — or that need the same spelling to mean two different things — hit AliasError::Conflict the moment the second registration is attempted. There is no merge operation to paper over it: separating the papers afterward means rebuilding the context from an export, not a small fix.
• Silent meaning drift. "Attention" or "fine-tuning" carries a different sense per paper, but inside one context it is one concept — resolve's gloss and every graph read blend facts from every paper that used the spelling. Nothing errors; the answers are just quietly wrong.
• Lost exclusion. Once merged, "answer from this paper only" cannot be expressed — a context is the finest unit a search can leave out, and nothing smaller exists inside it to leave out instead (Concepts). The fix is not a smarter query; it is not having merged the papers in the first place.

Other corpora draw other boundaries

The table in §1 generalizes. The paper's section-as-context, chapter-as-group mapping does not.

• Contracts: one contract is usually one context — its clauses share a vocabulary, and it is between contracts that the same term ("Term", "Party") most often needs to mean something different. A chapter-style group per contract is rarely worth it; a group over a contract family (a master agreement and its statements of work) is.
• Codebases: a module or crate is usually the context — identifiers are the vocabulary, and they are meant to mean one thing within it (see Concepts: source code); the repository is the group.
• Product manuals: a product version is usually the context, because the same feature name can mean something different across versions; the product line is the group that reaches across them.

Anti-patterns

No new machinery here either — every one of these is the responsibilities in §1, assigned to the wrong unit.

• Treating context as corpus by default: folding everything ingested in one sitting into one context. See the failure mode above.
• Writing a path into a source id and expecting a hierarchy: "paper/chapter/section" as a source id does not create groups, chapters, or sections — see the slash is typography.
• Promoting a token-limit chunk to a context boundary: a chunk is a pipeline transport unit with no server meaning — see the responsibilities table's last row.
• Building an artificial root group only to search everything: an ad-hoc cross-context question names its groups directly in the request; see cross-search.
• Reaching for a single-context retriever after splitting a corpus into contexts: the split only pays off if searches actually cross it with contexts/groups — see cross-search.