Troubleshooting — slow, stuck, or the wrong shape

Start here: symptom → layer → diagnostic → remedy

Find the row that matches, run the one diagnostic, then follow the remedy link. Every diagnostic in the table below is read-only.

None of the rows above are fixed by deleting the data directory. A wipe destroys the corpus along with whatever evidence would have shown which layer was actually at fault — and the fault is usually not in Taguru's stored data at all. Reach for a reset last, not first, and if a remedy crosses a release boundary, take a backup before you start (see below).

Version skew: reading every layer's version

Server, SDK, and integration package all ship from this one repository in lockstep — one version number per release, not one per package. The fix for skew is always the same: make every layer read the same minor version.

# server — a subcommand, not a flag
taguru version

# server, without a shell in the container (the image is FROM scratch)
curl -s localhost:8248/metrics | grep taguru_build_info
curl -s localhost:8248/health   # {"status": "ok", "version": "…"}
docker run --rm ghcr.io/t0k0sh1/taguru:0.9.0 version

# the wire-contract versions this server speaks (ADR 0005 §6) —
# the machine-readable answer, ahead of any SDK connecting
curl -s localhost:8248/version | jq

# Python SDK / LangChain integration
python -c "import taguru; print(taguru.__version__, taguru.SUPPORTED_HTTP_CONTRACTS)"
python -c "import taguru_langchain; print(taguru_langchain.__version__)"

# TypeScript SDK / LangChain integration
node -e "const t = require('taguru'); console.log(t.VERSION, t.SUPPORTED_HTTP_CONTRACTS)"
npm ls langchain-taguru

The MCP bridge answers the same question over the protocol: its initialize response's serverInfo.version names the server it's attached to, and the same instructions text carries the GET /version block below (folded into GET /protocol's own trailer) — an MCP client reads contract versions without a second connection.

GET /health's own success body carries a version too, per ADR 0002 §10 — {"status": "ok", "version": "…"} — for scripts that already poll it and would rather not add a second call. taguru router's own /health carries the same field beside its router/shards keys. A server built before this field existed (0.4.x and earlier) still answers the bare text ok with no version at all; the --url forms of import/export/ compact read this field once per run and warn on stderr (without blocking) when it names a different minor than the CLI itself.

Before 0.6.0, a minor version bump was allowed to change a response's shape without any machine-readable signal, and one already did: POST /contexts/{name}/sources/search (and the cross-context form) went from a bare array of hits to a PassagePage object, {plan, hits}, in 0.4.0. A 0.3.x SDK against a 0.4.x server failed with expected an object for PassagePage, got list; a 0.4.x SDK against a 0.3.x server failed with no hits key to find. Neither side had any way to detect the mismatch before that decode error.

ADR 0005 exists to make that class of failure loud and early instead. Server and SDK versions still ship in lockstep (one release, one number, across every package — server X.Y.Z pairs with SDK X.Y.Z), but the wire shape itself is now versioned independently as http_contract/mcp_contract (GET /version above), and both official SDKs check the server's http_contract.supported range against their own before their first real request:

The hits themselves were unchanged by the 0.4.0 break; they moved under a hits key, and plan was new — per-context lane execution (which of BM25/vector ran, why one didn't, the effective cosine floor). The remedy for a genuine mismatch is still never to patch around the shape: pin an older SDK to an older server (the error message gives the exact pip/npm command), or upgrade every layer together.

The Docker image you think you're running

image: ghcr.io/t0k0sh1/taguru:latest guarantees nothing after the first pull — Docker does not re-pull an already-cached tag on up or a plain restart.

Diagnostic: compare the running server's reported version against the release you expect. /metrics's taguru_build_info{version="…"} is the ground truth (the image has no shell for docker exec to fall back on); docker compose images taguru or docker inspect's digest are secondary checks against what's cached locally.

Update: pull, then recreate — a pulled image alone changes nothing until the container is rebuilt from it.

docker compose pull taguru
docker compose up -d

The durable fix is to stop using latest at all. deploy/docker-compose.yml pins on purpose (image: ghcr.io/t0k0sh1/taguru:0.9.0 # pin; latest moves) — see Docker Compose — the image for the pin and digest-pin guidance.

Back up before crossing a format bump. Image formats migrate forward on load and never write the old version back out — rolling the binary back past a format bump means rolling the volume back with it. See Docker Compose — the model for why a rollback is a restore, and Docker Compose — backups for POST /flush + snapshot, taguru export/import, and TAGURU_REPLICATE_URL + taguru restore.

Is compact worth running on this context?

The append-only image format never reclaims space on its own between compactions: retraction unlinks attribution records but doesn't erase them, and alias removal leaves arena bytes behind. compact --dry-run (issue #371) answers "how much dead weight is standing right now" without rebuilding anything, so you can decide whether a rebuild is worth it before paying for one.

# offline — reads state.directory(), opens nothing
taguru compact --dry-run

# a running server — GET /contexts, never POST .../compact
taguru compact --dry-run --url http://127.0.0.1:8248

# machine-readable, e.g. to gate a maintenance job on dead_ratio
taguru compact --dry-run --json | jq '.[] | select(.dead_ratio > 0.2)'

Each row reports dead_edges, dead_ratio, dead_attributions, and arena_slack — the exact numbers a real compaction would shed, since they're read from the same live-for-hot/snapshot-for-cold ContextStats GET /contexts already serves (no server-side change was needed to add this: the stats were already on the wire, just not surfaced by the CLI). One thing it deliberately does not predict: the rebuilt image's byte size (bytes_after) — that depends on exactly how the rewrite lays out the surviving content, which only running the rewrite produces. Whether that rebuilt image actually reached disk is a separate question from whether the numbers above are real: a real compaction's response also carries image_persisted — false means the rebuild itself succeeded (the graph in memory is already smaller) but the durable copy on disk is still the old, larger one, most plausibly because the disk was full at the moment compaction tried to reclaim space on it. The next flush tick retries the publish on its own — no need to re-run compact yourself — but that retry keeps failing for as long as the underlying cause does (check disk space first); once it's resolved, either watch the server log for the retry to stop warning, or re-run compact once more and confirm image_persisted: true in its response. A row with "stats_are_snapshot": true means the context is cold (not currently loaded) or came from a remote listing, so its numbers are the last-saved snapshot rather than a live recomputation — still accurate as of the last flush, just not up-to-the-millisecond.

There's no fixed threshold that's right for every deployment — a context with heavy revision traffic and light footprint may not be worth compacting even at a high dead_ratio, and TAGURU_AUTO_COMPACT already handles the common case (ratio-triggered, from the flusher tick) without any CLI involvement at all. This command is for the cases automatic compaction doesn't cover: an opted-out deployment, or deciding by hand whether a specific context is worth the maintenance window before running taguru compact for real.

Erasing a document completely (mis-ingest recovery, deletion requests)

Retraction withdraws a document's truth; compaction is what removes its bytes. For a deletion request (GDPR-style right to be forgotten) or a bad ingest you want gone entirely, you need both.

# 1. preview what the retraction would touch — writes nothing
curl -X POST 'localhost:8248/contexts/sake/sources/retract?dry_run=true' \
  -H 'Content-Type: application/json' -d '{"source": "docs/old.md"}'

# 2. retract: withdraw the source's associations and remove its passage
curl -X POST localhost:8248/contexts/sake/sources/retract \
  -H 'Content-Type: application/json' -d '{"source": "docs/old.md"}'

# 3. compact: physically drop the withdrawn records and the passage text's bytes
curl -X POST localhost:8248/contexts/sake/compact

Why the second step isn't enough by itself: storage is append-only on both sides. Retraction unlinks the source's attribution records from the graph but leaves them as dead space in the image, and removes its passage from serving while the text's bytes stay in the passage log behind a tombstone. POST /contexts/{name}/compact rebuilds the image without the dead records and rewrites the passage log without the retracted text (passages_compacted: true in its response confirms the log rewrite ran). Until a compaction runs — yours, or the automatic ratio/size-triggered ones — the withdrawn content is unreachable through every API but still present on disk. For an actual deletion-request obligation, also check image_persisted in the same response: the compaction endpoint answers 200 as soon as the rebuild is done in memory, but if the rebuilt image could not be written back to disk (a full disk being the likely cause, which is also exactly when a deletion is hardest to complete), the OLD image — with the shed bytes still in it — is what a crash right after would boot from. A deletion request is not complete until the response confirms image_persisted: true; image_persisted: false means resolve whatever kept it off disk (disk space, most plausibly) first, then re-run POST .../compact (safe to repeat) and check the response again.

Two boundaries to know. Names survive: concepts and relation labels the document minted stay interned in the image (they are shared vocabulary, not document content) — if a concept name is itself the sensitive datum, that name needs its own remediation beyond this runbook. Replicas replay the same history: the retraction replicates like any write, but each replica's own on-disk files hold their own dead bytes until a compaction runs there too — for a strict erasure deadline, compact the writer, then each replica (or restore replicas from a post-compaction snapshot).

Whose seconds are they: Taguru's or the model's?

Every request Taguru serves logs one line to stderr — read that line before assuming Taguru is the slow part.

The access-log middleware emits a tracing::info! line per request, message "http", carrying method, route, context, group, status, key, and latency_ms. With TAGURU_LOG_FORMAT=json it's one JSON object per request — the least ambiguous way to read it:

{"fields":{"message":"http","method":"POST","route":"/contexts/sake/sources/search","context":"sake","group":null,"status":200,"key":"-","latency_ms":6.2}}

The default, human-readable stderr format carries the same fields on one line per request, timestamp and level first. TAGURU_LOG_SEARCHES=1 adds a second, opt-in line per resolve/activate/sources/search call naming the cue, hit count, and top score — useful for judging relevance, though it doesn't carry latency itself.

Cues are the user's memory content. This event log is off by default on purpose — turning it on copies raw query text into your log pipeline, which is a data-handling decision, not just a debug toggle. Turn it on only when you've thought about where those logs land and who can read them, and turn it back off once you're done.

The procedure: time the full round trip at the client, then read latency_ms on the retrieval call inside that same window. The difference is everything downstream of Taguru — the integration code and the LLM's generation. As a rule of thumb, retrieval is milliseconds and local generation is seconds to minutes; an answer that takes 45 seconds with an 8 ms search line behind it is not a Taguru problem — see the serving layer.

One nuance: a slow latency_ms on sources/search can itself be the embedding provider — the query embedding's round trip happens inside that request. Cross-check the taguru_embedding_breaker_* series on /metrics before concluding Taguru's own retrieval path is at fault.

The semantic lane: 404s and silent degrade

Two different failure modes wear the same name ("embeddings aren't working") but look nothing alike in practice.

Loud failure — a configuration mistake. TAGURU_EMBED_URL must be the full OpenAI-compatible endpoint, not a base URL — http://127.0.0.1:8257/v1/embeddings, not http://127.0.0.1:8257. Point it at a base URL and the provider 404s; Taguru does not retry a 404 (only 429 and 5xx are treated as transient), so it fails the request immediately with embedding provider answered HTTP 404. Diagnostic: hit the configured URL directly.

curl -s -X POST "$TAGURU_EMBED_URL" \
  -H 'content-type: application/json' \
  -d '{"model":"'"$TAGURU_EMBED_MODEL"'","input":["ping"]}'

Quiet degrade — search keeps answering. A failure at query time (provider down, timeout, circuit breaker open after 3 consecutive failures) doesn't fail the search — it logs a warning, passage query embedding failed; serving the lexical lane alone, and answers from BM25 only. The response says so in-band, which is the check that actually distinguishes "results got worse" from "something broke":

{
  "plan": {
    "contexts": [
      { "name": "sake", "lanes": { "vector": { "ran": false, "reason": "the query embedding failed: …" } } }
    ]
  },
  "hits": [ … ]
}

sources/search/explain reports the same reason in more detail. A degraded result is never pinned into the retrieval cache, so the next request tries the embedding call again rather than repeating a stale lexical-only answer forever.

Why the symptom is "paraphrases stop matching" rather than "search breaks": BM25 always runs, and the vector lane is fused in by reciprocal rank fusion on top of it — losing the vector lane degrades recall on paraphrase and synonym queries, it doesn't empty the result set. See Internal architecture — search for the two lanes and fusion.

A third failure mode, quieter than either above: the right-language model, ranking badly. No 404, no degrade — the vector lane runs and reports success, and still ranks badly, because the embedding model itself doesn't separate the corpus's language well. Measured on a Japanese sake-brewery corpus: Ollama's nomic-embed-text reversed a resolve — cue "新潟の酒蔵" scored the correct brand (青嶺酒造) at 0.57 but a manufacturing-process term (洗米) at 0.65, ranking the process term above the entity the cue actually names. Point TAGURU_EMBED_MODEL at a model built for multilingual retrieval instead before assuming a floor problem — embeddinggemma through the same Ollama endpoint, or the in-process local provider's multilingual-e5-small / -base (see the local RAG walkthrough).

A calibrate OVERLAP verdict can be correct even on a healthy multilingual model. With the ranking order fixed, taguru calibrate still reported OVERLAP for embeddinggemma on the same corpus: the expected name's cosine band (0.40–0.62) overlapped the best other candidate's band (0.47–0.69). The cause here differs from the gloss-cross-reference case measured on Google Cloud — this corpus's concept-name glosses run 2–5 characters, too little text for the model to anchor a separation at any floor. A short gloss folding in more context (the label outline describe already returns, not just the bare name) is the lever that can help; a floor number alone cannot fix an overlapping band — calibrate reports it rather than papering over it, by design. See calibrate for what the verdict means.

The serving layer: thinking, context, caps, streaming

None of the four below are Taguru settings — they belong to the model server behind whatever base URL your LLM client points at. Each mechanism is general; the concrete knob shown is Ollama's, the server this was field-tested against.

• Thinking mode. A reasoning model can spend the entire time budget on invisible thinking tokens before the first byte of an actual answer — a busy GPU, then a timeout or, with no timeout set, an empty response after minutes. Disabling thinking is a decision made at the serving layer or the client integration, not at Taguru: Taguru's own LLM-facing tooling speaks plain OpenAI-compatible chat and never toggles a vendor's thinking flag. Either pick a non-thinking model or disable thinking where the model is served.
• Context window. An unspecified or too-small context window truncates a large prompt silently — no error, just a worse answer built on a partial prompt — and also changes the model's VRAM footprint and load latency, which is why the first request after an idle period can be far slower than the rest. On Ollama, the window is set per served model, not per request: bake it into a derived model with FROM <base> + PARAMETER num_ctx 16384, then ollama create.
• Output cap and prompt volume. Generation time scales with the output token cap (Ollama's num_predict), how many passages retrieval handed back, and the size of the assembled prompt. Trimming retrieval breadth (fewer hits, tighter top_k) is a legitimate way to cut generation latency, not just a relevance knob.
• One model for extraction, a different one for answers. A disciplined, structured-output-friendly model for extraction and a faster, more conversational one for answering the user are usually different picks — but alternating both through a single serving slot on one GPU pays a full reload on every swap. Give each its own slot, or its own served instance, if both run concurrently. The local RAG walkthrough keeps extract, embed, and answer as three separately configured local models end to end.
• Streaming changes what arrives first, not how long the whole thing takes. Streaming improves time-to-first-token — the user sees something sooner — but total generation time for a given output length is the same with or without it. A fast-feeling first token is not evidence of a fast request; measure the full completion, not the first chunk.

Seeing the split: spans, request IDs, token counts

The fastest way to stop re-diagnosing the same "is it Taguru or the model" question is to make the split visible in your own logs, once, rather than re-deriving it from raw timing every time.

Wrap the retrieval call and the generation call in separate spans rather than one "searching…" span that quietly includes the LLM call — a UI spinner that covers both stages is exactly what makes a slow generation look like a slow search. Record, per attempt: a request ID, the provider's finish_reason, input/output token counts, and elapsed time. That record is what turns "it's slow" into a specific layer the next time it happens.

Taguru ships the retrieval-side half of this natively: OTEL_EXPORTER_OTLP_ENDPOINT turns every request into a span tree, not one flat span — taguru.retrieve nests a phase span per step of the documented loop (resolve, describe, query, activate, citations, passage fallback), with the embedding provider call and the BM25/ANN/fuse lanes as further children, and a trace_id in the access log so logs and traces cross-reference. Full span-tree reference: Tracing. Taguru has no visibility into the answering call at all — it happens in your application, after retrieval has already handed back hits — so that half has to be instrumented on your side.

For the shape to mirror, Taguru's own extraction pipeline is a working example of attempt-level LLM observability, even though it's a different LLM call (building the corpus offline, not answering a query): taguru extract --diagnostics-out writes elapsed_seconds, finish_reason, and token counts per attempt as JSONL, and the LangChain SDK's on_event callback emits the same fields as typed events. That per-attempt shape — an id, finish_reason, token counts, elapsed time — is exactly what the answering call needs too.