Sparse retrieval architecture

Talqora’s sparse path is a native BM25 full-text retrieval capability. It is the lexical half of the retrieval system: use it for customer names, product names, policy language, dates, codes, titles, and other terms where the words themselves matter.

Sparse retrieval runs by itself with search_type: "sparse", or alongside semantic similarity with search_type: "hybrid".

When sparse text matters

Dense vectors are excellent at meaning, but a one-off identifier can be underweighted by semantic similarity. Include sparse_text whenever users may search for names, SKUs, policy identifiers, version strings, direct quotations, or uncommon terminology.

Write concise, searchable content rather than opaque serialized payloads. A product record should include its title, brand, category, SKU, and useful attributes. A document chunk should include the heading and extracted body. A support case should include the issue title, symptoms, resolution, and useful identifiers. Keep authorization fields in metadata filters, not as a substitute for access control in text.

Write path

When you send sparse_text, Talqora assigns an immutable document version, builds lexical retrieval data, and makes the version searchable before the write completes.

POST /vectors
  -> validate dimensions and metadata
  -> write the dense vector
  -> index versioned sparse text
  -> publish immutable retrieval data
  -> activate the version in Talqora's consistency ledger

The consistency ledger stores vector IDs, active versions, byte counts, and timestamps. It does not store customer document bodies.

Immutable versions and consistency

Lexical retrieval data is published as immutable versions rather than modifying a live posting list in place. A prior physical version can exist briefly after a record is replaced or deleted, but Talqora’s version ledger is the correctness boundary:

1. An upsert receives a new logical version.
2. The new sparse record is published before its version becomes active.
3. A query verifies every lexical candidate against the active ledger.
4. Old versions and tombstones are excluded immediately, before background compaction reclaims their storage.

This makes retries and asynchronous compaction safe without exposing a stale document to a customer query.

Query behavior

A sparse request supplies a lexical query:

{
  "search_type": "sparse",
  "sparse_query": "INC-1842 oauth timeout",
  "top_k": 10,
  "min_score": 0.15,
  "filter": {
    "tenant_id": {"$eq": "acme"}
  }
}

BM25 uses term frequency, inverse document frequency, field-length normalization, and term positions. Talqora applies supported metadata constraints, verifies candidate versions, and returns only records still live in the index. Scores rank results within this mode; they are not probabilities and should not be compared with scores from another retrieval mode.

Use a relevance threshold evaluated on your own query set. For an exact-lookup UI, a low threshold can be appropriate because the query is specific. For a user-facing assistant, use a stronger threshold and allow a no-evidence response rather than grounding an answer in a weak match.

Phrases, literals, and numbers

Quote a sparse query to request a phrase match, for example "vendor retention policy". Unquoted terms are a broad lexical query. For a copied email address, error code, URL, UUID, hash, or version identifier, use literal exact search. For a bounded pattern over original text, use regex search.

A bare number is usually weak evidence in a long, repetitive source. Combine it with a heading or identifier, apply a metadata filter such as page, or use a phrase. Confirm the returned snippet before presenting it as evidence to an end user.

Deletes and retries

A delete removes the active ledger row before physical retrieval cleanup is scheduled. The document therefore disappears from results even if storage cleanup is still running. Writes and deletes require Idempotency-Key, so a network retry can safely repeat the same operation.

Talqora retries transient retrieval-service responses with bounded exponential backoff. Permanent schema or query errors are returned without retrying. Callers should follow the same discipline: retry failed network operations with the same Idempotency-Key, respect 429 and Retry-After, and fix malformed payloads, invalid metadata filters, or dimension mismatches before retrying.

Isolation

Each Talqora index has an isolated lexical retrieval namespace. The only public surface is the Talqora Vector API, which applies API-key scopes, organization authorization, rate limits, active-version validation, metadata filters, and hybrid fusion before returning evidence.

Dense vectors and sparse text use independent retrieval paths. A dense-only record may omit sparse_text; sparse and hybrid queries consider only records with live indexed sparse text.

Operating a sparse workload

Measure sparse retrieval with the same rigor as dense retrieval. Track query latency, candidate count, empty-result rate, queried transfer, and relevance of exact identifiers. Inspect source-text quality when codes are missing before changing rank parameters; poor lexical results often come from absent or noisy sparse_text.

For hybrid workloads, compare dense-only, sparse-only, and hybrid result sets on a representative evaluation corpus. This shows whether an issue is semantic representation, lexical source quality, a missing metadata filter, or a threshold that hides valid evidence. See Hybrid retrieval for rank-fusion behavior.