Deploy on Kubernetes

The model you are deploying

One fsync-then-apply owner per data directory is what makes "200 = durable" simple enough to trust. The advisory lock exists to enforce that, not to be worked around — plan the deployment on this model rather than against it.

• Deploys are stop-then-start. The downtime is the boot: cold registration is cheap however many contexts exist, and pinned contexts preload in parallel — the port opens when they finish. Never let the platform run two pods against one volume, even during a rollout.
• Availability is promotion time with a replica pool; restore time without one. There is still no leader election or automatic promotion — by design. Without replication, the recovery unit is the data directory: a storage-layer snapshot restores byte-exactly, an export stream restores anywhere. With TAGURU_REPLICATE_URL set, the bucket carries the lineage continuously (seconds of RPO), an empty volume boots straight from it (the stateless variant) — and a read replica is the warm standby: losing the writer costs the manual promotion walk (drain the lag metric to zero, start a writer with stated intent, flip the name) plus the dead writer's un-shipped tail, which the replica's /metrics had on display the whole time. Rehearse whichever rung is the plan — the promotion drill is automated as an integration test server-side, but your fleet's version of it is yours to time.
• Scale reads with replicas; scale writes by sharding behind one router. Read traffic — and LLM-memory traffic is read-dominant — fans out across a replica pool tailing the same bucket, with no coordination protocol. Write throughput scales by giving independent writers disjoint sets of contexts and putting taguru router in front: one Service for clients, groups and cross-context search spanning every shard with the single-instance merge semantics — contexts no longer need to cohabit to be searched together. What does not work is two instances sharing one data directory: on local disks the lock refuses it, and on NFS/EFS the lock may fail to.
• A rollback is a restore. Image formats migrate forward on load and never write the old version back out, so rolling the binary back past a format bump needs the data directory rolled back with it (snapshot) or re-imported from an export stream. Check the release notes for format bumps before downgrading.

The same model on a single host, without a scheduler in the loop, is the Docker Compose page.

Installing

Credentials stay out of the manifest — the Deployment reads them from a Secret.

kubectl create namespace taguru
kubectl -n taguru create secret generic taguru-keys \
  --from-literal=TAGURU_API_TOKENS='ops:CHANGE-ME'
kubectl -n taguru apply -f kubernetes.yaml

Named keys (name:token) mean the access log says which key was used and a leak costs one revocation. Grow the rest of the configuration in the Secret or a ConfigMap as the deployment grows (TAGURU_RATE_LIMIT_PER_MIN, the TAGURU_EMBED_* tier, cache budgets, …) — every knob is an environment variable, listed by taguru --help.

Prefer kubectl apply -k? deploy/kustomize packages these same manifests: the raw files as verbatim bases, overlays for the stateless / replicas / router variants, and the retuned knobs (image tag, storage, resources, probe budgets) as documented patches — its README records why kustomize over a Helm chart, and CI — on every change under deploy/ — keeps the in-tree manifest copies byte-identical to the raw files, schema-validates every rendered configuration, and keeps the base render-equivalent to kubernetes.yaml.

The load-bearing choices

All of them follow from one data directory owning durability.

• replicas: 1 and strategy: Recreate. The advisory lock refuses a second writer on local disks and cannot be trusted to on shared network filesystems, so the platform must never run two pods against one volume. Recreate makes every rollout stop-then-start; the default RollingUpdate would briefly run old and new side by side.
• ReadWriteOnce, block-backed storage (EBS/PD/Ceph RBD) — never ReadWriteMany/NFS. Single-attachment enforced by the platform is what keeps the single-writer model honest where the lock can't.
• Three probes, three different questions:

  Probe	Endpoint	Meaning
  startupProbe	/live	Boot = cold registration + parallel pinned preload; the port opens when it finishes. Sized periodSeconds: 5 × failureThreshold: 60 = five minutes of grace — resize to your fleet.
  livenessProbe	/live	Restarts fix wedged processes, not disks — so liveness must not watch the write path.
  readinessProbe	/health	503 while the write path is degraded (a failing flush). Routing away is the remedy; restarting is not. Self-heals within one flush interval once the disk recovers.

• fsGroup: 65532 makes the volume writable by the scratch image's uid, and readOnlyRootFilesystem: true holds because /data is the only write target. The image is the server binary alone on scratch — no shell to kubectl exec into; the observability surface is /metrics, the logs, and the probes — plus, with OTEL_EXPORTER_OTLP_ENDPOINT pointed at a collector Service, a full retrieval span tree (see Tracing) that needs no shell access to read.
• Memory requests ≈ TAGURU_CACHE_BYTES (default 512 MiB) + pinned contexts + headroom (the manifest ships 1Gi request / 2Gi limit). Don't guess at scale — taguru estimate --associations N builds a context of the target size and measures it.
• The Service stays inside the cluster. Expose beyond it only behind TLS (Ingress/Gateway with a certificate) — a bearer token is the whole credential — and set TAGURU_RATE_LIMIT_PER_MIN when you do.
• The image version is pinned (latest moves): with formats that migrate forward only, a surprise binary bump on a pod reschedule is a one-way door.

The stateless variant: the volume becomes a cache

deploy/kubernetes-stateless.yaml makes the bucket the source of truth: emptyDir instead of the PVC, and the pod hydrates from TAGURU_REPLICATE_URL at boot.

With replication on, an empty data directory is not a problem to refuse — it is a node that has not downloaded its state yet. The stateless pod boots by hydrating: shared files (groups, the grant store, every context's sidecar meta) land first so the whole directory is enumerable, pinned contexts download in parallel before the port opens (the same readiness shape as the ordinary preload — size the startupProbe as pinned bytes ÷ bucket bandwidth), and everything else hydrates on first touch or via a background fill. Local files whose bytes already match the bucket's manifest are reused without a download, so a pod that kept its emptyDir across a container restart comes back cheap. Instance loss stops being volume surgery: any node starts empty and serves.

What this variant trades away, stated plainly: RPO becomes the shipping lag for real. A crashed pod's un-shipped tail (seconds at the default cadence) lived only in its emptyDir; the PVC variant's WAL replay would have recovered it, this one loses it. And because every fresh pod starts empty, the manifest bakes in TAGURU_TAKEOVER=1: after a crash the replacement must depose its predecessor's generation — still fresh by heartbeat, never cleanly retired — or it would refuse to start for the 300-second takeover grace. That acknowledgment is the politeness layer only; epoch fencing remains the arbiter, so even a misconfigured duplicate Deployment cannot corrupt the lineage — but it will silently depose the writer you meant to keep. Never point two deployments at one bucket prefix.

On the PVC variant the same machinery is the recovery story rather than the boot story: a rescheduled pod reuses its volume untouched (its writer holds the bucket's newest generation, so nothing is re-verified), and starting a writer against a bucket whose owner still looks alive demands serve --take-over / TAGURU_TAKEOVER=1 — starting a writer against a bucket IS the promotion act.

The read pool: replicas tail the bucket

deploy/kubernetes-replicas.yaml adds a horizontally scaled read Deployment (TAGURU_REPLICA=1, emptyDir, replicas: N, RollingUpdate allowed) behind its own taguru-read Service, beside either writer manifest.

A replica is the stateless boot running forever: hydrate from the bucket, then keep tailing its manifest — new writes land within the writer's shipping lag plus the poll interval (TAGURU_REPLICATE_INTERVAL_MS, one knob for both roles). Every retrieval verb serves from the replica's own copy, so reads scale with the pool; every mutating verb — the raw HTTP routes and the MCP write tools alike — answers 403 read_only_replica naming the writer (TAGURU_WRITER_URL, the writer Service's name), so a misdirected client fails crisply instead of retrying into a loop. The topology stays boring on purpose: writes go to the writer's Service, reads to taguru-read, discovery is DNS — no cluster protocol, no gossip.

What the pool promises, honestly: each context is consistent at its own applied watermark (a shipped-cycle boundary — never a torn read), but two contexts can sit at different cycles, and staleness is bounded by shipping lag + poll interval. A bucket outage freezes replicas at their last watermark — they keep serving, /health stays green (routing away a whole healthy pool over a bucket blip would be worse), and the staleness shows on /metrics: taguru_replica_applied_seq vs taguru_replica_shipped_seq per context and lane, taguru_replica_behind_seconds, and the manifest/poll timestamps. Those same gauges are the promotion-time RPO on display — the number the manual promotion runbook reads before a replica's directory (or any empty one) is restarted as the next writer. Auth note: give the pool the writer's TAGURU_API_TOKENS secret; OAuth delegation stays at the writer (grants are minted there, and a replica refuses the grant endpoints like any other write).

Sharding: one router in front of many writers

taguru router (issue #130) turns manual sharding into one front door: a stateless Deployment — scale it freely, RollingUpdate, no volume — whose only config is a context→shard map file mounted from a ConfigMap. The router kustomize overlay is this section worked out: two writer shards, the route-map ConfigMap (content-hashed, so a map edit rolls the routers on apply — the belt over the router's own hot reload below), and the front-door Deployment and Service.

The map is boring on purpose: context = shard-url per line (point each URL at that shard's writer Service), plus an optional * = shard-url fallback for contexts the map does not name. Set TAGURU_ROUTE_MAP to the mounted path. The map hot-reloads exactly like the auth table (issue #515): SIGHUP, or the file's own content-digest watch (~5s — it sees the kubelet's atomic symlink swap, so a plain non-hashed ConfigMap propagates by itself, no roll), with a broken edit refused whole while the old map keeps serving (taguru_router_map_reloads_total counts both outcomes). Context verbs proxy to the owning shard byte-for-byte; cross-context recall/query/sources/search and groups span every shard with the exact single-instance merge semantics (cursors forward verbatim — they anchor on the last match, not on a per-instance position), and POST /mcp works unchanged, so taguru-mcp and claude.ai-style clients point at the router's Service like a single server. The router holds no keys: it forwards Authorization and the shards enforce it, so give every shard the same token Secret. OAuth (TAGURU_PUBLIC_URL) stays off the router — consents are durable state a stateless fleet cannot hold.

Failure honesty: a shard that answers an error fails the request whole (exactly as one failing context does on a single instance); a shard that cannot be reached degrades the fan-out reads to labeled partials — the response gains an unreached array naming the shard and its contexts — and routed verbs answer 502 shard_unreachable, retryable by design. Per-shard outcomes are on the router's /metrics (taguru_router_shard_requests_total). Moving a context between shards is a runbook, not a protocol — in order: quiesce its writes → taguru export → DELETE it through the router (the old shard drops it, group projections included) → edit the ConfigMap → re-import through the router, which now routes it to the new shard. Delete before re-import: a leftover copy keeps answering the old shard's slice of every group fan-out. And let the reload land everywhere before re-importing — a router still holding the old map would route the stream back to the old shard (watch the reload counter, or roll the fleet if you want a hard barrier). Each shard can keep its own bucket (TAGURU_REPLICATE_URL) and read pool — the rungs compose.

Backups and restore

Continuous replication to object storage is the primary path; volume snapshots remain for point-in-time copies.

Set TAGURU_REPLICATE_URL (s3://bucket/prefix, GCS and Azure Blob likewise, credentials via the pod's usual identity — IRSA, Workload Identity) and the server ships every file family to the bucket continuously: RPO is the shipping lag (seconds, exported at /metrics) instead of the snapshot interval, and DR degrades to the bucket's own cross-region replication. Recovery is taguru restore --out /data into a fresh volume, then taguru inspect — or no step at all: a server started on an empty volume with the same URL boots straight from the bucket (see the stateless variant). The bucket is epoch-fenced: if two live instances ever point at one URL (a botched restore, a duplicated deployment), the newer claim wins and the older server's shipper fail-stops loudly (taguru_replication_fenced, plus a taguru::audit line) while the server itself keeps serving its local data.

The snapshot procedure, unchanged, for point-in-time copies and format-rollback insurance:

1. POST /flush — write out everything dirty.
2. Snapshot the PersistentVolume at the storage layer (a CSI VolumeSnapshot, or the backing store's own snapshot — every writer is fsync+rename, so a point-in-time snapshot is safe at any instant). Or take the portable stream with GET /contexts/{name}/export and restore anywhere through POST /import.
3. Verify the restored copy with taguru inspect — the same fully-validating load + WAL replay the server does. Verification runs anywhere the image runs: mount the restored volume on any machine and docker run --rm -v …:/data ghcr.io/t0k0sh1/taguru inspect /data.

Back up each context's file family as a set, always — copying files out of a live volume does not guarantee cross-file consistency; snapshot, replicate, or stop the writer. And rehearse the restore: on this model, the restore drill is the availability story. Details in operational basics.