Ingest connectors — taguru_langchain.ingest_connectors

The Python shape, not a CLI

There is no taguru connectors … subcommand and no connector flag on taguru extract — this is an SDK import, not a binary.

from taguru_langchain import FilesystemCheckpointStore
from taguru_langchain.ingest_connectors import sync_references

report = sync_references(
    ["docs/manual.md", "docs/manual.pdf", "https://example.com/notes.html"],
    ingester=ingester,
    checkpoints=FilesystemCheckpointStore(".taguru-checkpoints"),
    events_out="sync.jsonl",
)
print(report.to_dict())  # one JSON object: counts, duration_ms, an events_path reference

ADR 0007 §3 weighed three packaging shapes: a second Rust binary, a Python SDK module, or a protocol plus a Python reference implementation. It chose the third — connector parsing stays entirely client-side, exactly as the SDK's own .md/.txt reference connector already worked, rather than adding a PDF/HTML/DOCX/PPTX parser dependency (and the corresponding Rust ecosystem immaturity, Cargo.lock audit surface, and scratch-image bloat) to src/. Every connector commit (#347–#353) states "no change to src/, the HTTP contract, or the MCP contract" for this reason — the server-side batch/import contract this page's connectors feed is unchanged; only the producer is new.

The connectors

Every format connector implements the same Connector protocol: a .read(reference) -> ConnectorDocument method, plus parser/ parser_version/parse_options_digest for checkpointing. None of them raise on an ordinary parse failure — a document with problems comes back with empty text and populated diagnostics instead (see below).

Format-specific parser dependencies are opt-in extras, not the default install — a caller ingesting nothing but .md/.txt never pays for a PDF parser it never uses. Constructing a connector whose extra is not installed raises ImportError with the exact pip install line to run.

• PdfConnector. One {"kind": "page", "value": ...} locator per paragraph, derived from the PDF's own page boundaries; its outline (bookmarks), if any, becomes sections and the document title. A page whose extracted text has fewer than min_chars_per_page (default 16) non-whitespace characters is ocr_required rather than silently passed through as low-quality text. Encrypted and corrupt PDFs are reported the same structured way, never a raised exception.
• HtmlConnector. Boilerplate (script/style/nav/aside, and a page's own header/footer when nothing scopes content to a <main>/<article>) is stripped before text is built. The heading hierarchy survives as a breadcrumb section ("Guide > Installation"); each heading's own id (or its nearest id-bearing ancestor's) becomes a fragment locator on every paragraph up to the next heading. A URL fetch's source id is the final, fragment-stripped, canonicalized URL — <link rel="canonical">, when present, only ever populates metadata.canonical_url, never the source id. By default a fetch also refuses any destination (including one reached only via a redirect) that resolves to a private, loopback, link-local, or multicast address; pass allow_private_networks=True to fetch one intentionally.
• DocxConnector. The document body is walked in real document order — paragraphs and tables interleaved, never read as two separate collections. A table (top-level or nested inside another table's cell) becomes exactly one paragraph carrying a table locator; an ordinary body paragraph never carries a locator, which is what makes "this paragraph has a locator" mean "this paragraph is a table." A password-protected .docx is recognized by its container's own signature and reported encrypted before ever being opened as a zip. Footnotes, endnotes, comments, and text-box content are named in a single partial_extraction diagnostic rather than silently short-changed. Only .docx is read; .doc and .docm are both unsupported_format.
• PptxConnector. A slide's shapes are walked in document order, recursing into group shapes. Every non-empty text-frame paragraph and every table carries a slide locator; a slide's speaker notes are read as their own paragraph(s) carrying speaker_notes instead. A slide's title becomes the paragraph-anchored section. Charts, SmartArt, and embedded/linked OLE objects are named in a single partial_extraction diagnostic. Only .pptx is read; .ppt and .pptm are both unsupported_format.
• S3Connector/sync_object_storage. Lists a bucket/prefix and dispatches each object to whichever connector above its extension (or, failing that, its content-type) names, then re-stamps the delegate's source/ metadata.origin_uri/display_name to the object's own identity. The store behind it is pluggable: open_object_store speaks the same scheme set the server's replication path does — s3://, gs:// (GCSObjectStore), az:// (AzureBlobObjectStore), and file:// (stdlib-only, the test/air-gapped backend). GCS lists a generation for every object whether or not the bucket has versioning enabled, so its checkpoint fingerprint always sits on the strongest tier; Azure's blob index tags map onto object tags exactly as S3's do, and GCS stands its custom metadata in for them. See the credential boundary and checkpoints below for what makes a rerun cheap.
• watch_directory. The continuous form of the same sync, for a local tree: run one sync_object_storage pass over the directory, yield its RunReport, wait the interval, repeat — a generator, so the caller owns the loop and break (or a should_stop event, honored before, during, and between passes) ends the watch. Polling by design, not inotify/FSEvents: the checkpoint already skips an unchanged file on its (size, mtime) listing fingerprint without opening it, so a pass over a quiet tree costs one directory walk, and no platform-specific event API (which network mounts and bind mounts break anyway) enters the dependency tree. Deletion stays report-only by default, exactly as every sync pass.

The credential boundary survives

The same posture document extraction holds for chat model credentials applies here to storage credentials.

• Object storage: each cloud's standard credential chain only. S3ObjectStore's only construction knobs are endpoint_url, region_name, and profile_name; GCSObjectStore's is project; AzureBlobObjectStore's are account (the account name) and endpoint_url — plus, on all three, a pre-built client escape hatch for callers that construct the SDK client themselves. With client, credential configuration and custody are entirely the caller's: whatever that client was built with is what runs, and the guarantees below apply only to the internally-constructed path. None of the named knobs is credential material, and there is deliberately no parameter any of the three could accept an access key, a service-account key, or a SAS token through. Credentials always come from the cloud's own chain — boto3's (environment, shared config, an EC2/ECS/Lambda role, …), Application Default Credentials, or DefaultAzureCredential (environment, workload/managed identity, Azure CLI, …). An endpoint_url with userinfo embedded (https://user:pass@host) is rejected outright.
• URLs: canonicalization is the only representation. HtmlConnector fetches always strip userinfo and a deny-listed set of query parameters (signature, sig, token, access_token, apikey, api_key; the full AWS SigV4 presign set x-amz-signature, x-amz-credential, x-amz-security-token, x-amz-date, x-amz-expires, x-amz-algorithm, x-amz-signedheaders — the per-issuance companions too, or every fresh presign of the same object would mint a new source id; and the GCS V4 signed-URL equivalents x-goog-signature, x-goog-credential, x-goog-date, x-goog-expires, x-goog-algorithm, x-goog-signedheaders) before the result becomes a source id, a checkpoint key, a batch's source, a log line, or an observability event — the same one value serves every purpose; there is no separate "redacted display value" to keep in sync with it.
• Auth failures are permanent, never retried. Object-storage errors classify into two buckets: access-denied/invalid-credential/expired-token/no-such-bucket style codes are permanent (raised immediately, no retry), everything else is transient. Status codes are not trusted alone for this — every store answers 404 for a missing bucket/container too — so the error code decides where the cloud names one (S3, Azure's ContainerNotFound vs BlobNotFound), and the call site decides where it doesn't (GCS: a 404 from a listing is the bucket, from a fetch it's the object).

What a connector produces

A connector never numbers its own paragraphs — it produces text and lets the same splitter document extraction uses decide where paragraphs begin and end, so a connector's locators/sections stay addressed against whatever the server's own canonical paragraph numbering turns out to be. ConnectorDocument.sections/.locators round-trip losslessly through /import into Citation.section/.locator on every recall/explore/activate/cite_passage response — see import.html's locator lines.

fingerprint_inputs.raw_content_sha256 hashes the object's raw bytes before parsing — a different question from taguru extract's manifest, which hashes the extracted text. The two are intentionally not interchangeable: one answers "did the object change," the other "did the parsed content change."

Citation locators, by connector

A locator ({kind, value}) names the exact paragraph it was recorded for — unlike section, it does not extend to the next paragraph. kind is an open string; the standard connectors populate it as follows:

A notable asymmetry: DocxConnector gives an ordinary body paragraph no locator at all, so "this paragraph has a locator" means "this paragraph is a table." A DOCX has no page-like structure of its own to spend a locator budget on instead; a PPTX slide already has a number, so its budget goes to distinguishing slide body from speaker notes rather than to tables.

Work units and size limits

The work unit is one file or one object — one ConnectorDocument, one source id, one retract-then-apply unit — the same granularity as taguru extract's own work units. A connector does not split a PDF into per-page or per-section documents on its own; page/slide boundaries surface as locators within one document, not as separate source ids. sub_source_id() exposes a grammar for a finer-grained id (manual.pdf#installation) for a runner that wants to split a document itself before feeding it through, but the standard connectors do not call it.

Model-input chunking (how much text one LLM call sees) is a separate, later layer — TaguruIngester's own chunking runs after a connector hands back one document's full text, and is independent of any locator a connector emitted. Three size ceilings apply before that:

• Raw bytes. max_file_bytes (default 64 MiB for PDF/DOCX/PPTX/S3, 16 MiB for HTML) rejects an oversized source as content_too_large before parsing it.
• S3 listing size. S3Connector checks an object's listed size against the same ceiling before fetching it at all.
• Extracted text. A parsed document's own text is capped separately (8 MiB) — content_too_large again if parsing itself produced more text than that.

Checkpoints: no new format

A connector's own fetch/parse work is independently resumable, composing with — never replacing — the same CheckpointStore chunk checkpoints already use.

from taguru_langchain.ingest_connectors import ConnectorCheckpoint

checkpoint = ConnectorCheckpoint(
    FilesystemCheckpointStore(".taguru-checkpoints"), namespace="connector"
)
cached = checkpoint.load(source, fresh_fingerprint)  # None on any mismatch — never a false hit

There is no dedicated connector manifest file or directory — every layer keys into the same CheckpointStore under its own namespace: connector: (a whole ConnectorDocument, keyed by source), s3-object: (a single object's fingerprint gate), s3-inventory: (a prefix's last-seen object list, for deletion detection), and file-probe: (a local file's cheap --dry-run stat). A missing, corrupt, or fingerprint-mismatched checkpoint always degrades to "unseen, redo the work" — never to a false hit. ConnectorCheckpoint.load additionally refuses a checkpoint whose stored source doesn't match the one being loaded, so two distinct sources that happen to fetch byte-identical content can't swap checkpoints under a buggy or lossy custom store.

For S3, checkpoint fingerprints prefer, in order: version_id > checksum > (size, last_modified) > a bare ETag as the last resort — some S3-compatible stores compute an ETag in a way that isn't a reliable content hash for multipart uploads, so it is never the first choice. An object whose listing metadata is unchanged is never even fetched; one whose bytes are unchanged despite a metadata bump (a tag edit, a copy-in-place) is fetched but never re-ingested — two independent skip gates, both fail toward re-doing the work.

Deleted objects are never retracted by default: report.deleted_detected names them, but nothing changes in taguru until you opt in — deletion_policy="retract" withdraws exactly what this connector's own prior listing no longer sees; "mirror" also reconciles against the context's actual source list, so it self-heals even on a first run with no prior listing to diff against.

Run report and event log

sync_references and sync_object_storage both return the same RunReport shape (ADR 0007 §11) — one event/summary vocabulary across every driver.

Every source moves through some subset of seven phases: discovered → unchanged | parsed → extracted → imported, or skipped / failed off that path. report.events is the full per-source phase history (SourceEvent: source, phase, elapsed_ms, bytes, parser, diagnostic); pass events_out= a path to stream it as JSONL as the run happens — one line per phase transition, flushed immediately, written even under dry_run=True since the sidecar is a dry run's whole product.

Counts tally each source's LAST phase only — never every phase it passed through. A run summary reading {"discovered": 0, "imported": 2} is normal, not a bug: both sources reached discovered on the way to imported, and only the final phase is counted. report.events/the JSONL sidecar is where the full transition history lives if you need it.

Two more details worth knowing before writing a JSONL consumer:

• Identify duplicates by diagnostic.code, not by source. A rejected duplicate reference is recorded as a skipped/duplicate_source event but never touches the summary tally — grouping by source instead can conflate a duplicate-input case with a genuinely interrupted-and-resumed run, since sync_object_storage's own duplicate-key case has reference == source.
• elapsed_ms is per-source, not per-run. It measures elapsed time since that source's own first event, not since the run started; discovered is always 0.0. bytes is the raw object's own size, not the parsed text's byte length.

report.to_dict() is a single JSON object — counts, duration_ms, interrupted, and an events field that is the event count (an int, not the array) plus an events_path reference, so the summary stays O(1) regardless of corpus size.

Two different dry runs

A driver's own dry_run=True is stricter than TaguruIngester.ingest_text's dry_run (which still calls the model and only skips import_batches). At the driver level, dry_run=True means no network fetch, no local file read beyond a cheap stat, and no write to the corpus or to either checkpoint store — this driver-level flag is never forwarded into the connector bridge; a real (non-dry) call always runs with the bridge's own dry_run=False.

Diagnostics

A closed vocabulary of nine codes, mirroring the shape of taguru extract's own diagnostics sidecar: unreadable, unsupported_format, encrypted, corrupt, ocr_required, source_id_too_long, content_too_large, partial_extraction, duplicate_source. A connector's .read() never raises for an ordinary parse problem — it returns a ConnectorDocument with empty text and one or more Diagnostic entries (code, message, source) instead, the same "never a silently empty passage" posture extraction takes. New codes may be added over time; an existing code's meaning is never repurposed.

OCR is a boundary, not an engine

No OCR engine ships in this package or in any connector (ADR 0007 §10). OcrAdapter is the external boundary a connector calls out to when one is configured — given the raw document bytes and the locators naming which pages/units are unusable, an adapter returns whatever text it could recover, each unit still tagged with the locator it was asked about. PdfConnector is the one connector wired to call an adapter today (ocr_adapter=), and only for the exact pages its own min_chars_per_page threshold found unusable — never the whole document. An adapter's own failure (an exception, or simply recovering nothing) leaves that page exactly ocr_required, as if no adapter had been configured at all.

from taguru_langchain.ingest_connectors import PdfConnector

document = PdfConnector(ocr_adapter=MyOcrAdapter()).read("docs/scanned.pdf")

Where to go next