Tracing — the composed retrieval loop, made legible

The span tree

Every Taguru-owned span is named taguru.*. The HTTP server span is the one exception, kept as semconv's {method} {route} — that's what already crosses process boundaries via traceparent, and giving it a second name would only make dashboards built on the semconv convention stop matching.

POST /mcp                                  # server span, otel.kind=server
└─ taguru.retrieve                         # root: one call composed the whole loop
   ├─ taguru.resolve
   ├─ taguru.describe                      # absent if describe_first=false
   ├─ taguru.query                         # absent unless `labels` pins facets
   ├─ taguru.activate
   ├─ taguru.citations                     # absent if fetch_citations=false
   │  └─ taguru.skip (event)               # reason=citation_passage_missing — one aggregate event carrying the miss count, not one per miss
   └─ taguru.passage_fallback              # absent unless the text lane actually ran
      └─ taguru.passage_search             # same span search_passages emits directly
         ├─ taguru.embed (client)          # query embedding, exactly one per search — cue-cache hits skip it; the ANN sweep receives the pre-embedded cue
         ├─ taguru.search.bm25
         ├─ taguru.search.ann              # absent: cache hit, or vector lane didn't run
         └─ taguru.search.fuse

taguru.passage_search is also the whole answer for a cache hit: one childless span, taguru.cache.result=hit, no lanes underneath — the absence of children is the signal, not a separate flag to check. A cross-context search (POST /sources/search) emits the same taguru.passage_search span once for the whole fan-out (taguru.context.count marks it), with one taguru.passage_search.target child per target — indexed by taguru.target.index, never the context name — each parenting that target's own lane spans.

assemble_evidence composes its own tree under taguru.assemble_evidence: the same resolve / query (labels-gated) / activate / citations phases, but no taguru.describe, the unconditional passage lane under its own name taguru.passages (this endpoint's passage search is a first-class lane, not a fallback), and — opt-in — taguru.communities.

Two more span kinds sit outside the retrieval tree proper:

• taguru.shard_call (client, otel.name overridden to "{method} -> shard {n}") — the router's dispatch to a shard, fan-out and transparent per-context proxy hops alike (#696), parenting that shard's own {method} {route} span. A router deployment gets end-to-end traces across every shard, not just its own hop.
• taguru.tool_call (server) — the stdio bridge's (taguru-mcp) per-job span, parented from the caller's traceparent when the MCP client sends one in params._meta (optional; a client that sends nothing behaves byte-identically).

Attributes and events

Every attribute value is bound to an existing metrics enum's as_str() — no parallel spelling grows out of sync with /metrics' own labels.

Count and size attributes ride alongside the enum-valued ones above, all exported as real numbers (i64 / f64): the taguru.*.count family (origin / label / anchor / association / activation / context), taguru.citation.requested / .returned / .missing, taguru.passage.hit_count and the per-lane hit split, taguru.limit, taguru.search.terms / .pool / .rows / .hits / .lexical_pool / .semantic_pool / .floor, taguru.rerank.candidates, taguru.embed.inputs, taguru.dispatch.bytes / taguru.result.bytes, and taguru.shard.index / taguru.target.index. The only identity strings are taguru.rerank.model and taguru.embed.model / .purpose — operator configuration, never user data. ADR 0008 §6 is the binding registry.

Three span-event names carry a taguru.reason field instead of a free-form message — tracing-opentelemetry maps a log line's message field to the OTel event name, so the stable code has to live in its own field to stay greppable across a fleet:

• taguru.skip — a planned step didn't run (or, aggregated, partially didn't). Reasons include describe_disabled, no_anchors, labels_absent, citations_disabled, citation_passage_missing (the aggregate citation-miss event; carries the count as taguru.citation.missing), budget_exhausted (the composed result outgrew TAGURU_MCP_MAX_RESULT_BYTES mid-loop), fallback_not_requested, fallback_suppressed, zero_limit, no_query_terms, origins_empty, communities_disabled, no_communities_artifact, deadline_exceeded_before_start (a retrieve refused whole because the budget was spent before any step ran).
• taguru.degrade — a step ran, but in a reduced form. Reasons are vector_off, vector_query_embedding_failed, vector_no_vectors, vector_model_changed, vector_width_changed (every vector-lane outcome except ran — a lane that ran has nothing to confess), plus bridge_unreachable — the stdio bridge answering a tool call with a transport error instead of the fronted server's reply.
• taguru.cache — a cache tier decided the result. Reasons include retrieval_cache_hit, semantic_cache_hit / _stale / _guarded / _miss, cue_cache_hit.

One producer sits outside those three event names: a batch import whose schema record cannot be applied warns with taguru.reason=schema_load_failed / schema_write_failed on its own import schema record failed event — the same stable-code discipline, a different event name.

Error semantics

A degraded-but-completed retrieval is not the same failure mode as one that returned nothing, and the two must not collide into one red span.

• otel.status_code=ERROR is set only by the span whose own operation failed to produce a result — it does not propagate to ancestors. A retrieval that degraded (semantic lane down, BM25 still answered) leaves every span UNSET, the OTel convention for success.
• .with_error_events_to_status(false) closes the one path that used to break this: tracing-opentelemetry special-cases a log field literally named error into an exception event and an automatic ERROR status. A tracing::warn!(error = …) on a successful degrade used to paint the whole request span red for no operational reason — fixed structurally by this one setting, not by renaming every occurrence (span-event fields are never named error going forward; see ADR 0008 §9).
• The HTTP server span keeps its existing rule (5xx → ERROR, 4xx does not) — a JSON-RPC tool error rides on a 200, so POST /mcp stays UNSET while taguru.retrieve goes ERROR. That split is the distinction #224 asked for.

Privacy

Enforced three ways, none of them a reviewer's judgment call at write time.

• No dynamic attribute keys, anywhere. tracing::info_span! and tracing::info! require field names at macro-expansion time — the full set of exportable keys is enumerable with grep, and no call site can invent a new one at runtime that carries a raw string.
• A target-level firewall on the export path. init_telemetry's OTel layer filters on tracing_subscriber::filter::Targets, with taguru::search forced OFF regardless of level — the one target TAGURU_LOG_SEARCHES=1 turns on for raw-query debug logging. Turning that env var on for local debugging can never leak a question into OTLP, even by accident.
• A sentinel integration test. A concept name, a source id, passage text, and a query each get a unique nonce; a full retrieval runs with TAGURU_LOG_SEARCHES=1 and OTLP export both on, and the test asserts none of the four nonces appear anywhere in the raw bytes the collector received.

Per-item data follows one rule: default to aggregate, never one span or event per item — not one event per citation miss (one taguru.skip event whose taguru.citation.missing attribute carries the count), not one span per resolved cue, not a context name anywhere (only counts — consistent with taguru_searches_total not carrying a context label either). The one exception is a router's shard list: shard identifiers are operator-controlled configuration, not caller-shaped data, so taguru.shard_call is one span per shard.

Context propagation

Every hop both extracts an inbound parent and injects its own current span outbound — this was previously extract-only.

• HTTP: inbound W3C traceparent / tracestate (and AWS X-Amzn-Trace-Id as a fallback) become the request span's parent, as before. New: every outbound call this process makes — router → shard, SDK → server — injects its own current span, not a bare pass-through of whatever header it received. A router fanning out to N shards now produces N child spans under its own request span, not one flat trace per shard.
• stdio (taguru-mcp): MCP has no HTTP headers to carry a traceparent on, so it rides in params._meta.traceparent / .tracestate — optional, read-only, and a client that sends nothing behaves exactly as before. The bridge injects its own span the same way on every outbound call to the server it fronts.
• SDKs (Python/TypeScript): inject on every outbound request, the same way — see below.

SDK tracing (opt-in, zero required dependencies)

retrieve() composes the identical client-side loop the server's own composed retrieve (taguru.retrieve, src/mcp/retrieve.rs) does — resolve → describe → query → activate → citations → passage fallback — under its own taguru.retrieve root, with the same taguru.skip reason vocabulary. sdk/spec/tracing.yaml pins the exact span names, attribute keys, and reason codes both SDKs share; each language's own test suite loads it and asserts against it directly.

Neither SDK's required dependency set grows. Both add OpenTelemetry as strictly optional:

• Python — pip install "taguru[otel]" pulls in opentelemetry-api; without it, every call in taguru._tracing degrades to a silent no-op (a plain pip install taguru is unaffected either way). Spans record only through three privacy-safe setters — a bounded int, a bool, or a closed reason code — never a free-form string.
• TypeScript — @opentelemetry/api is an optional peer dependency (peerDependenciesMeta.optional: true), loaded through a lazily cached dynamic import() so a plain npm install taguru never attempts to resolve it and the build (tsup --external @opentelemetry/api) never bundles it.
• Neither SDK opens a client-side HTTP span for the request itself — the server's own request span already covers that round trip as a child of whichever phase span made the call, so a client span in between would duplicate it without adding information.
• Tracing activates only once the application configures a TracerProvider — neither SDK calls set_tracer_provider / setGlobalTracerProvider itself. Wire up whatever OTel SDK you already use; these two just emit spans into it.

Try it locally: Jaeger via Docker Compose

No changes to your server config beyond one environment variable.

# one-off Jaeger all-in-one, OTLP/HTTP on 4318, UI on 16686
docker run -d --name jaeger \
  -p 16686:16686 -p 4318:4318 \
  jaegertracing/all-in-one:latest

# point Taguru at it and start the server (see docker-compose.html for the
# full compose walkthrough — add this one variable to that file's `environment:`)
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 \
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
  taguru serve

Run a retrieve tool call through any MCP client (or curl -X POST localhost:8248/mcp, see GET /protocol), then open localhost:16686 and search for the taguru service. One trace should show taguru.retrieve with its phase spans nested underneath, matching the tree above — with any lane that degraded or any step that was skipped visible as an event, reason code included.

To correlate with logs: TAGURU_LOG_FORMAT=json puts trace_id on every access-log line for a traced request. Grep the log for the trace id you care about, then paste it straight into Jaeger's trace-lookup search — no separate correlation system needed.

Sampled-out is not the same as disabled. With export configured but a sampler set to drop a given trace, tracing-opentelemetry still builds the span (and the SDK still runs the sampler) — the cost just never reaches the network. True zero cost only happens with OTEL_EXPORTER_OTLP_ENDPOINT unset entirely: every taguru::trace::span! call short-circuits to Span::none() without even opening the tracing registry's storage.