Records

Building retrieval that works

Retrieval is how you give an AI model the right context at the right moment: pull the handful of documents that answer a question, feed them to a Prompt step or an Agent, and let the model reason over facts instead of guessing. The reflex is to reach for a vector database immediately. Often you should not. This guide covers how to right-size retrieval, when to add vector search, and how to prove it works, using Records, the vector Flow steps, and the Semantic Search tool.

Right-size before you reach for vectors

A vector database is powerful and it is also overhead: embeddings to generate, an index to keep fresh, chunks to tune. For a lot of real problems you do not need it.

  • Small corpus, simple lookups. If you have a few hundred documents and queries map to a field you can filter on, use Records with a list-records or get-record step and a recordFilter. Structured filtering over a small, well-named set is faster, cheaper, and easier to debug than semantic search, and it never returns a plausible-but-wrong neighbor.
  • The whole corpus fits in context. If the relevant material is small enough to hand the model directly, do that. Stuffing a few documents into a Prompt step beats standing up retrieval infrastructure to fetch what you could have passed inline.
  • Keyword lookup is enough. A keyword or web search behind a tool (Exa, for example) is a legitimate first version with a clean upgrade path. You can add vector search later, when you can show it helps.

Reach for vector search when the corpus is large, the queries are phrased in the user’s words rather than the document’s, and meaning matters more than exact tokens. That is the case it was built for. Before that point, simpler retrieval usually wins.

Hybrid search is the default for real documents

When you do go to vector search, pure semantic similarity is rarely the right default for enterprise documents. Embeddings are excellent at meaning and surprisingly bad at exact tokens: part numbers, error codes, dates, proper nouns, acronyms, and jargon are precisely where a keyword match beats a vector match, and precisely what business questions hinge on.

Hybrid search combines vector similarity with keyword matching, and it is the safer default for document corpora. On Runtype, hybrid search is a capability of the vector provider you choose for the Semantic Search tool. Weaviate supports it; the built-in pgvector store and some other providers are pure vector. If your users search by exact identifiers as often as by concept, pick a provider that does hybrid, or keep a keyword path alongside your vector path. See Weaviate (vector search) for provider setup.

Filter with metadata, do not filter after

The biggest precision win in retrieval is usually not a better embedding model. It is metadata filtering.

Extract structured metadata when you ingest a document (its type, source, date, owner, product area, whatever your queries scope by) and store it alongside the vector. Then filter into the search: constrain the candidate set before similarity ranking, so the search only ever considers the right slice. Filtering after the search, by fetching a broad set and discarding the wrong ones in a later step, is slower and quietly drops relevant results when the discarded ones crowded out the good ones.

For Records-backed retrieval, this is exactly what recordFilter does: name your Records so a lookup keys straight to the slice you want (doc_{{product}}_{{version}}), and the filter runs in the query. For vector stores, providers like Weaviate expose metadata filtering on the search itself. Either way, the principle holds: narrow first, rank second.

Chunk with structure, not with a fixed ruler

How you split documents for embedding decides what retrieval can find. A few defaults hold up well:

  • Aim for roughly 200 to 300 tokens per chunk, split on sentence boundaries so you never cut a sentence in half.
  • Overlap adjacent chunks by a sentence or two so a fact that straddles a boundary survives in at least one chunk.
  • Prefer the document’s natural structure over a blind character count. Split on headings, sections, or list items when the document has them; a chunk that is one coherent section retrieves far better than one that is an arbitrary 300-token window.
  • Normalize entities between extraction and any grouping step. If one chunk says “Acme Inc.” and another says “Acme”, a later aggregate treats them as two things. Canonicalize names at ingest.

Runtype does not impose a chunker: you shape chunks in the ingest Flow (a Prompt step or a transform-data step that splits the source), generate embeddings with a generate-embedding step, and write them with a store-vector step. Because you own that Flow, chunking is a design choice, not a black box, so it is worth getting right.

Add a rerank and a query-rewrite step when precision matters

Two composition patterns sharpen retrieval without a new primitive:

  • Rerank. Vector search gives you the top candidates by similarity, which is a coarse signal. When precision matters, retrieve a slightly wider set, then add a Prompt step with a cheap, fast model that scores each candidate against the query and keeps the best few. A small reranker on top of a broad retrieval is one of the highest-return additions you can make.
  • Query rewrite or decomposition. A raw user question is often a poor search query. For multi-part or vague questions, put a Prompt step before the search that rewrites the query, or breaks a compound question into separate searches whose results you merge. Searching for a clean query beats searching for whatever the user typed.

Keep the retrieved context minimal

More retrieved context is not better context. Two failure modes make this concrete:

  • Distractors actively hurt. A chunk that is similar to the query but wrong does not just waste space; it pulls the model toward a wrong answer. Tightening your top-K (returning fewer, better results) and de-duplicating near-identical chunks often improves generation more than any retrieval tweak, because you are removing the confusing neighbors.
  • A big context window does not make broad retrieval safe. Models degrade as their context fills with marginally relevant material, even when everything technically fits. “Retrieve more and let the long context sort it out” is not a strategy. Retrieve the minimal set that answers the question and stop.

The vector-search step’s limit and threshold are your controls here. Start tight and loosen only if you can show recall is suffering.

Prove it with two eval suites

Retrieval quality and generation quality are different things, and lumping them into one score hides which one is broken. A wrong final answer can come from retrieving the wrong chunk or from the model fumbling the right chunk. Build two Eval suites so you can tell them apart:

  1. Retrieval quality. Fix a set of queries with the chunks that should come back, and check that the search returns them. This measures the search in isolation, with no model in the loop.
  2. Generation quality given correct context. Hand the model the known-correct chunks and check that it produces a good answer. This measures the generation in isolation, with retrieval held constant.

To seed these, bootstrap synthetic query-to-chunk pairs: for each chunk, generate a question it answers, and deliberately paraphrase and vary those questions so they do not just reuse the document’s wording (real users never phrase things the way the document does). Include distractor cases: queries where a similar-but-wrong chunk should not be returned. And do not trust a public embedding leaderboard over your own results. A model that tops a benchmark can still lose to a simpler one on your documents; a golden set of 50 to 90 real judgments from your own corpus is worth more than any leaderboard rank.

One shift in what you measure: once retrieval feeds an Agent rather than a human reading a results list, stop optimizing ranking metrics and start measuring the end task. What matters is whether the Agent answered correctly and how many tokens it burned getting there, not whether the ideal chunk was ranked first or third.

Keeping a knowledge base fresh

For a corpus that changes, retrieval quality decays as the index drifts from the source. Use a Schedule to re-run your ingest Flow on a cadence, re-embedding new and changed documents so the index stays current. One record or chunk per document, keyed by a stable name, makes it cheap to re-ingest a single item without rebuilding the whole index.

One caution on exotic stores: a specialized datastore is not automatically a retrieval upgrade. “Graph RAG” is a retrieval technique, not a mandate to run a graph database, and most teams reaching for one would be better served by hybrid search plus good metadata filtering. Add specialized infrastructure only when a simpler design has demonstrably hit its limit on your evals.

Next steps