Taguru
how it works · internals

Internal architecture — disk is the truth, memory is a cache

From the "one context = one flat buffer" decision, through WAL durability, the three retrieval paths, auth, and observability. Each design decision is also written down as a module comment in the corresponding source — this page is the map.

The map

clients
LLM agentsClaude Code / Desktop, Converse loops, …
HTTP clientscurl, pipelines, monitoring
claude.ai connectorsremote MCP via OAuth
surface
taguru-mcpstdio bridge; translates tool calls into HTTP
HTTP APIaxum. Every endpoint + POST /import
POST /mcpStreamable HTTP (stateless profile)
cross-cutting
authnamed Bearer keys
oauthfor remote MCP; built-in AS
limitstime budgets and rates
metrics / tracePrometheus · OTLP
registry
registrywhole-context cache (LRU + pinning), dirty tracking, periodic flusher, cold-boot registration
per context
Contextthe association graph itself (flat buffer)
passagesoriginal-text store (snapshot + append log)
bm25paragraph inverted index (derived; rebuildable)
vectorsgloss / paragraph embeddings (optional)
disk
{name}.ctx and the rest of the file family+ .wal.jsonl / oauth.json / .taguru.lock — all written fsync + rename

The library layer — one context = one flat buffer

context.rs. The invariant that all state fits in one flat buffer is what keeps everything below it simple.

  • State is "a UTF-8 string arena + seven tables of fixed-width #[repr(C)] records". Adjacency lists are intrusive chains threaded through the edge records, so pointer chasing allocates nothing.
  • Every mutation is an append or a field update. So the whole state round-trips as one image through to_bytes/from_bytes — little-endian, fully validated on load, closed by a CRC-32C footer so bit-rot cannot load as truth.
  • Capacity is the u32 space: ~4.29 billion records per table, 4 GiB of interned text. Overflow is a ContextFull error, not a panic (507 over HTTP, and the write was not applied).
  • The retrieval entrance is normalization (NFKC, case folding, katakana→hiragana) plus a bigram inverted index. Aliases are entry-only alternative spellings; results always carry the canonical spelling.
recallqueryquery_anydescribeexploreactivateresolveresolve_labelunreachable_from associateassociate_fromadd_concept_aliasadd_label_alias

Indigo = reads, plain = writes. That is the entire library surface; everything else is the server's concern.

The server layer — the registry and the file family

registry.rs. Disk is the source of truth; memory is a cache managed at whole-context granularity.

data/
├── {name}.ctx                 # the graph image (raw bytes of to_bytes)
├── {name}.meta.json           # description / pinning / stats sidecar (kept outside so the image stays a pure dump)
├── {name}.wal.jsonl           # graph write WAL (truncated after every successful flush)
├── {name}.passages.bin        # compacted snapshot of the original text
├── {name}.passages.wal.jsonl  # append log of the original text
├── {name}.bm25.bin            # paragraph inverted index (derived — lose it and it rebuilds; never an outage)
├── {name}.vectors.bin         # embedding cache (keyed by model name)
├── {name}.group               # one group's record (name, description, members, children) — fsync + rename
├── oauth.json                 # OAuth grants for remote MCP
└── .taguru.lock               # advisory lock on the data directory
  • Boot registers every context cold (only pinned ones preload). The first access loads transparently; past the cache budget (TAGURU_CACHE_BYTES) least-recently-used contexts are evicted. Pinning is the floor of that policy; TAGURU_CONTEXT_QUOTAS (issue #136) adds the ceiling side — a context past its declared cache_bytes share is evicted before any compliant one under pressure (no reservation while there is slack), and its declared storage_bytes makes every growth entrance — graph batches, passage stores, /import batch by batch — refuse with 507 storage_full once the on-disk family reaches it, while retract/compact/delete stay open as the ways back under. The storage gate reads the same numbers the per-context gauges serve: the live WAL lanes plus the flush-refreshed snapshot, so enforcement and observability cannot disagree. Compaction's own rebuild can shrink the graph in memory even at the ceiling — reaching the ceiling is exactly when its DISK write is most likely to fail, so its response carries image_persisted to say whether the smaller image actually landed, distinct from the call succeeding at all.
  • A write only marks its context dirty. Persistence happens on the periodic flusher (TAGURU_FLUSH_SECS), on eviction, and on shutdown.
  • Every directory row carries revision counters {graph, passages, config} (issue #149) — applied graph writes, the passage log watermark, and config/embedding changes respectively — the "has anything changed since I last looked" token a retrieval cache keys on, with a group-level fingerprint hashing the member contexts' counters. The honest contract: within one process every read is live and strictly monotonic; across a clean shutdown the persisted values are exact; across a crash a cold context can serve a lagging value until its first load catches the graph counter up against the WAL replay (the same posture as the cold stats snapshot — and the search paths load before computing, so a cache fill never keys on the stale seed). Compare for equality only; a cache that outlives the process must treat a server restart or a delete-recreate as invalidation.
  • The exact-match retrieval cache (issue #150) is those counters' first in-process consumer: an identical recall/query/passage-search request (cross variants included, and MCP tool calls, which dispatch onto the same routes) against an unchanged corpus answers from the stored response bytes without re-running the search. Invalidation IS the key — each key carries, per resolved target, the pair of revision lanes that surface depends on (recall/query: graph+passages, for section enrichment; passage search: passages+config, for published vectors and the context floor) plus a per-incarnation identity nonce, all read before the search runs — so a bumped lane simply makes old entries unreachable, and a delete-recreate, a replica lineage switch, or a compaction (which drops what the revision counter alone can't express — retracted edges and orphaned aliases stop appearing in query results even though nothing was written) changes the nonce, and there is no purge hook and no TTL anywhere. Scope is materialized into the resolved target list, so two credentials share an entry exactly when their grants resolve a request identically. Byte-budgeted tick-LRU (TAGURU_RETRIEVAL_CACHE_BYTES, default 32 MiB, 0 = off); hits replay the served-response metrics so taguru_searches_total and the lane-contribution counters read continuously, while the hit/miss split lives in taguru_retrieval_cache_total.
  • The semantic cache tier (issue #153, passage search only, off unless TAGURU_SEMANTIC_CACHE_THRESHOLD is set) stores no payloads and invalidates nothing: it holds only equivalence claims — "this query asks what that earlier query asked", proven by query-vs-query embedding cosine over the threshold AND a text guard finding no negation/number/entity mismatch (cosine alone routinely conflates a question with its negation). A claim that holds rewrites the request's exact-cache key to the canonical query's parameters under the request's own current fingerprints and serves those bytes — so freshness rides entirely on the exact tier's revision lanes and identity nonce (a write turns the claim's serve into a stale fall-through, and the fresh fill re-canonicalizes the cluster), and a semantic serve never contaminates the exact tier's "identical is literal" contract. The query embedding shares the search's own cue cache, so the fresh path still pays exactly one provider call. Outcomes land in taguru_semantic_cache_total{outcome="hit"|"stale"|"guarded"|"miss"}guarded is the tuning signal — with a claim-count gauge beside it.
  • Every search response carries its execution plan (issue #151): the contexts actually consulted in effective order (for the cross variants, the resolved target list — groups expanded, grants applied), and — for passage search, per context — whether each lane ran, the reason when one was skipped (embeddings off / nothing embedded yet / model changed / provider refused, the same prose the explain endpoint uses), the effective cosine floor when the vector lane swept, and — when the request carried a source filter (issues #167/#169) — a filter: {eligible_sources, total_sources} block naming how many of the context's sources were eligible before either lane ran. The plan lives inside the result, so the retrieval caches replay it byte-identically with the hits it accounts for — coherent by construction, because every event that could change a plan (a corpus write, a vector publish, a floor change, a different filter) also moves the cache key. The one transient state that recovers without a revision bump — a refused query embedding — is therefore never cached at all: the degraded BM25-only page is served but not filled, so a provider blip is not pinned until the next unrelated write.
  • The truth of one context is its whole file family — back it up as a set, always (operational basics).
  • Groups ({name}.group) stay resident whole and reconcile at boot: dangling members are dropped, an unparseable file is set aside as {name}.group.corrupt and reset empty. A group record reaches only the contexts of its own data directory — on a standalone instance a group cannot span deployments; behind taguru router every shard holds the group with its member list projected by the map, and the router unions the projections (fingerprints folded) back into the whole group — see the router bullet.

Durability — the WAL is what makes "200 = durable"

wal.rs · passages.rs. The flush interval is freshness cadence, not a loss window.

  • A graph change accepted by the HTTP API is appended to the WAL before memory is touched (JSON Lines, fsynced per batch). After a crash, the next boot replays it back.
  • Replay is driven by sequence numbers alone: every record carries a monotonic seq, and the image header carries a "burned in up to here" watermark. Double-apply and dropped records are structurally impossible.
  • The WAL truncates after every successful flush. It only nears its ceiling (default 256 MiB) when flushes keep failing — past it, writes are refused with 500: better to say "cannot write" early than to grow the log forever.
  • Original text has its own durability path: a compacted snapshot plus an always-on append log. The predecessor rewrote one whole file every time, so importing N documents wrote O(N²) bytes — now it is the same "snapshot + log" shape as the graph.
  • Every writer is fsync + rename. A filesystem snapshot is safe at any instant, and taguru inspect verifies a backup with the same fully-validating, checksum-verifying load + WAL replay the server does (images, passage snapshots, and WAL records all carry CRC-32C).
  • Durability is two honest tiers once TAGURU_REPLICATE_URL is set (ship.rs): a background shipper polls the data directory and continuously copies every file family — both log lanes tailed record-by-record, published files whole — to object storage (S3 / GCS / Azure Blob / file://). Tier one, a local crash: still loses nothing — the claim above is unchanged, and shipping adds zero work to the acknowledge path (it polls; nothing signals it). Tier two, losing the machine or the volume: a restore from the bucket loses at most the shipping lag — seconds, exported per lane at /metrics. taguru restore --out DIR materializes a directory from the bucket; the derived sidecars (BM25, vectors) ride along as a restore-cost optimization but are rebuildable, so restore tolerates their absence.
  • The bucket is epoch-fenced. The flock guards one local directory, not the bucket — a botched restore or a doubled deployment can put two live writers behind one URL. Each writer claims a monotonic generation with a conditional create and ships only into its own gen-N/ namespace; a deposed writer's shipper fail-stops loudly (latched metric + taguru::audit line) while its serve path keeps answering from local truth. No TTL, no automatic failover — by design; the fence is lease-compatible (a permanent lease with TTL 0) should automation ever be layered on. A per-generation liveness heartbeat and a clean-shutdown marker exist purely as ergonomics for the next writer's takeover guard below — never as an arbiter.
  • An empty directory boots from the bucket (hydrate.rs, issue #128): with the URL set, a server started on nothing materializes the newest complete generation lazily — shared files and every context's sidecar meta before boot, pinned contexts in parallel before the port opens, the rest on first touch or via a background fill. The complete marker carries a manifest (every shipped extent's length + CRC-32C), so local files that already match are reused without a download and every downloaded byte is verified; the successor's own generation is not marked complete until every family settles locally, so a restore can never land on a hollow lineage. Because boot-from-bucket removes the volume as the physical mutex, deposing a generation that still looks alive (fresh heartbeat, no clean shutdown) demands the operator's stated intent — serve --take-over / TAGURU_TAKEOVER=1; starting a writer against a bucket IS the promotion act. The deposed writer's un-shipped tail exists only on its own volume, and a successor hydrating elsewhere serves the lineage without it — the takeover's honestly-stated cost.
  • Read replicas tail the shipped stream (replica.rs, issue #129): serve --replica is that same hydration running forever — poll the bucket's manifest, re-verify what moved, land it, drop the loaded copy so the next read replays through the ordinary load path. A replica never claims a generation and never ships; every mutating verb (HTTP and the MCP write tools alike) answers 403 read_only_replica naming the writer (TAGURU_WRITER_URL, plus the bucket's fence holder), so no client retry loop can form. LLM-memory traffic is read-dominant and every retrieval verb serves from the replica's own copy, so reads scale with the pool — and the writer model is untouched. The consistency statement, honestly: consistent per context at that context's applied watermark; cross-context skew is possible; staleness ≤ shipping lag + poll interval; a bucket outage freezes the replica at its last watermark (it keeps serving, and /metrics says how stale). Per-context RPO is on display as taguru_replica_applied_seq vs taguru_replica_shipped_seq and taguru_replica_behind_seconds.
  • Promotion is manual, and it is a restart — the runbook (rehearsed end-to-end by the promotion integration test):
    1. Stop the old writer — or accept that the fence will cut its shipping the moment a successor claims.
    2. Drain: watch the standby's taguru_replica_behind_seconds reach 0 against a fresh taguru_replica_manifest_timestamp_seconds. What you see behind here is what promotion will lose.
    3. Claim: start a writer against the bucket — the standby's own directory restarted without --replica (its cache re-verifies warm), or any empty directory. A crashed predecessor guards the claim: state the intent with --take-over / TAGURU_TAKEOVER=1; a cleanly stopped one never asks.
    4. Flip the name: point the writer's DNS/Service at the new process. Replicas re-aim at the new generation by themselves — no restarts.
    5. Say what was lost: the dead writer's acknowledged-but-unshipped tail. That is the async-replication RPO — step 2 is where it was on display, and only synchronous shipping (not leases, not auto-failover) would have changed it.
  • Sharding gets one front door (route.rs, issue #130): taguru router is a stateless scatter-gather router over independent writers, config = one context = shard-url map file (TAGURU_ROUTE_MAP, optional * = fallback). Context verbs proxy byte-for-byte to the owning shard; cross-context recall/query/sources/search fan out and merge with the exact single-instance semantics — the graph verbs re-rank with the same comparator (one weight scale, context/subject/label/object tiebreak), the passage verb re-interleaves by per-context rank, and the after cursor forwards to every shard verbatim because it anchors on the last match itself, not on any per-instance position. Groups exist on every shard with member lists projected by the map (child-group edges broadcast whole, so nesting verdicts cannot differ and closure commutes with projection); /import splits its stream by each batch's context and dry-run-preflights batch chunks and projected group records alike, so a stream one instance would refuse with nothing applied is refused the same way here. Equivalence — router over split shards ≡ one instance with the same contexts, cursors and refusals included — is pinned by an integration test. Failure honesty: a shard that answers an error fails the request whole (as one failing context does on a single instance); a shard that cannot be reached degrades fan-out reads to labeled partials (unreached in the envelope) and refuses routed verbs with 502 shard_unreachable. The router holds no data directory, no lock, no keys (auth forwards; shards enforce — keep keyrings identical) — run any number behind one LB. Moving a context, in order: quiesce its writes → taguru export → DELETE through the router (the old shard drops it, and sweeps it from its group projections) → map edit + rolling router restart → re-import through the router, which now routes it to the new shard. Delete before re-import, or the leftover copy keeps answering the old shard's slice of every group fan-out; and finish the restart before re-importing — a router still on the old map would route the stream back to the old shard.

One MCP surface

mcp.rs · remote_mcp.rs · src/bin/taguru-mcp.rs.

The tool definitions, the tool→HTTP-request mapping, and the JSON-RPC plumbing are written once, in mcp.rs, and shared by the stdio bridge and POST /mcp. The transports differ only in "how a translated request is executed" (the bridge does a ureq round trip; in-server it calls the same-process Router) and "how the reply travels". The remote side is the stateless profile: initialize hands out the playbook and every tool call stands alone — no SSE, no session IDs, nothing to resume.

Authentication, authorization, limits

auth.rs · oauth.rs · limits.rs.

  • Bearer is a named keyring: TAGURU_API_TOKEN (key name "default") and TAGURU_API_TOKENS ("ci:tokA,laptop:tokB"). Access logs say which key; revocation is per key, rotation is overlap (add the new key → migrate → remove the old). Invalid configuration refuses to boot — and the whole table (scopes included) hot-reloads (issue #134): SIGHUP, or editing the --config file (~5s watch), swaps it on the running server. Fail closed — a broken edit keeps the previous table, and a reload can never disarm auth ("tokens configured" → "no tokens" is refused). Every request resolves authentication and scope from one keyring snapshot, so a mid-request reload can neither tear the table nor let a just-removed key fall through to the unscoped admin default; OAuth delegations from a removed key die with it on their next request. One taguru::audit line per reload (names added/removed/rotated/rescoped, never token bytes), counted by taguru_keyring_reloads_total / _reload_refusals_total.
  • OAuth exists for remote MCP: RFC 9728 resource metadata + a built-in minimal authorization server (PKCE and dynamic registration included). "Logging in" is possession of an existing API key — paste the key on the consent page and that connection acts as that key. Tokens are stored SHA-256-hashed; grants live in data_dir/oauth.json (delete + restart to revoke; auto-expiry after 30 days unused). OAuth opens /mcp only.
  • Limits are layered: whole-request time budgets and per-key rates in limits.rs (a one-minute burst, then convergence to the steady rate; excess gets 429 with Retry-After). A global in-flight ceiling sheds general overload, while a smaller shared semaphore caps concurrent vocabulary audits and context compactions — including the flusher's own ratio-triggered auto-compaction (TAGURU_AUTO_COMPACT, at most one context per tick), which takes a permit from this same pool rather than adding a second ceiling — so whole-context sweeps cannot occupy the worker pool; refused requests answer 503 + Retry-After without queueing, and a refused auto-compaction just waits for a later tick. The body cap sits in the body layer; parameter caps sit in each handler (association batches 10,000 / list inputs 1,000 / every limit ceilinged at 1,000).
  • Brute-force defense: failed Bearer attempts are limited per source IP per minute (default 10). A correct token is never throttled. X-Forwarded-For is not trusted — behind a reverse proxy everyone shares the proxy's IP, so throttle at the proxy there.
  • Scoped credentials stop at /mcp, for now (#62): OAuth's resource metadata only ever names /mcp (see above), so a plain HTTP integrator has nothing to reach for but the long-lived, whole-keyring TAGURU_API_TOKENS — no scoped, expiring credential exists for direct HTTP callers. Left as-is deliberately: it's not yet clear who calls plain HTTP directly rather than through the SDKs, which cover most current usage, and that answer should drive the design rather than the other way around. If it becomes necessary, the two options on the table are widening the OAuth resource beyond /mcp, or a simple admin-issued short-lived-token endpoint. No code changed for this decision; recorded so the trade-off doesn't need re-deriving next time.
  • No CORS, on purpose (#62): nothing in the router sends Access-Control-Allow-Origin or answers a preflight. OAuth's consent screen is server-rendered — no cross-origin browser fetch to allow — and every shipped client (the Python/TypeScript SDKs, the MCP bridge) is a server-side or CLI agent, not a browser page. If a browser-based integration ever needs this, the planned shape is an opt-in allowlist (TAGURU_CORS_ORIGINS), never a wildcard default. No code changed for this decision.

Observability

metrics.rs · trace.rs. Hand-rolled RED — a few atomics and one render function don't need a facade crate.

  • GET /metrics: per-route request counts and latency histograms, cache/flush/WAL/embedding outcomes, a 500-cause breakdown (taguru_errors_total{kind=…}), search hits/misses (taguru_searches_total), the retrieval cache's hit/miss split per op (taguru_retrieval_cache_total) with entry/byte gauges, the semantic tier's outcome split (taguru_semantic_cache_total), the resolve tier breakdown (taguru_resolves_total — a rising semantic share means cues are drifting from the vocabulary), schema pre-write checks by outcome (taguru_schema_checks_total{outcome="ok"|"warned"|"refused"}, counted only at the entrances a schema actually gates — never a dry-run or the audit/validate diagnostics), the embedding circuit breaker's state and short-circuit counts (taguru_embedding_breaker_state / _consecutive_failures / _opened_total / _short_circuits_total, present once a provider is configured), WAL and passage-log size gauges, and the time of the last successful flush.
  • Context names never become metric labels uninvited: clients mint the names, so the moment they land in labels the series grow without bound. Per-context numbers live in the usage stats of GET /contexts (reads, misses, writes, last access) — advisory, persisted with each flush and on graceful shutdown; a read never causes a disk write. The two opt-ins are deliberate: replication/replica lag is per context by nature (a stuck context needs naming), and TAGURU_METRICS_PER_CONTEXT=1|all|N adds the taguru_context_* capacity families — on-disk bytes by file family (image, WAL lanes, passages, sidecars), resident bytes, pinned, concept/association/label/source counts, and per-context schema violations (taguru_context_schema_violations_total), with N bounding the fleet to its top-N by disk size. Disk sizes refresh at each flush (and POST /flush), never at scrape time — a scrape does not walk the data directory.
  • Distributed tracing is opt-in: set OTEL_EXPORTER_OTLP_ENDPOINT and every request becomes an OTLP span, joining incoming W3C traceparent / AWS X-Amzn-Trace-Id (ALB, API Gateway) traces — and every outbound call this process makes (router → shard, SDK → server) now injects its own current span in turn, not a bare header pass-through. The composed retrieval loop is a full span tree, not one flat request span: taguru.retrieve nests a phase span per step (resolve/describe/query/activate/citations/passage fallback), and passage search nests further into BM25/ANN/fuse lane spans, with degrade/skip/cache decisions recorded as events carrying a stable reason code — never a raw string, and never at the cost of coloring a degraded-but-successful request ERROR. The access log carries trace_id so logs and traces cross-reference. Any OTLP backend works — switching is collector configuration, not a Taguru change. Full span-tree reference, the attribute/event vocabulary, and a local Jaeger walkthrough: Tracing.

Shipping and sizing

  • The Docker image is one binary on scratch: statically linked with musl, ~13 MB. No shell, no libc, no CA bundle (TLS roots are baked into the binary). The HEALTHCHECK is taguru health — the binary's self-probe. uid 65532, --read-only capable.
  • Sizing is measured, not guessed: taguru estimate --associations 1_000_000 actually builds a context of the target size and measures it — the shape knobs are --concepts (default associations/2, floor 2), --labels (50), --sources (associations/20, floor 1), --name-bytes (24, per interned name), --embedding-dims (0 = semantic tier off; 3072 = text-embedding-3-large), and --passage-bytes (0 = no registered passage text). For latency, cargo run --release --example benchmark (per-operation timings at 100k/1M records; each example under examples/ has its own README).

The primary sources for each decision are the module comments: registry.rs (disk as truth), wal.rs (seq and watermark), passages.rs (escaping O(N²)), bm25.rs (its standing as a derived artifact), paragraph.rs (the single split function), embedding.rs (why glosses are embedded), mcp.rs (written once), oauth.rs (the built-in-AS tradeoff), limits.rs · metrics.rs · auth.rs.