RAG Retrieval Returns the Wrong Chunks: A Production Debugging Guide

When a RAG system that demoed well starts returning irrelevant chunks in production, the cause is almost never the embedding model. In order of how often I've found them: the approximate index is tuned for speed and quietly dropping the right result, the chunking splits answers across boundaries, the query depends on an exact token that dense vectors handle poorly, or the right passage is retrieved and then buried in the middle of a long prompt. Work through those four before you touch the model.

First: prove it is actually retrieval

The single most valuable thing you can add to a production RAG system is a log line containing the retrieved chunk IDs, their scores, and the final answer — stored together, queryable, for every request.

Without it, every debugging session is guesswork. With it, triage takes thirty seconds:

• The correct passage is in the retrieved set, and the answer is still wrong → generation or prompt problem. Skip to the section on position.
• The correct passage is not in the retrieved set → retrieval problem. Continue below.

Build a small labelled set while you're at it — fifty real user queries with the document that should answer each one. You do not need a formal eval harness on day one. You need to be able to answer "did the right chunk come back, yes or no" on a fixed set of queries, so that when you change something you know whether it helped.

Cause 1: your approximate index is trading recall for speed

This is the one that explains "it worked in my notebook." In a notebook you probably had a few thousand vectors and did an exact scan. In production you added an index, because sequential scans over millions of rows are not viable.

pgvector's README is unusually direct about the consequence: "Unlike typical indexes, you will see different results for queries after adding an approximate index." Both of its index types expose a recall knob, and both default conservatively:

An ivfflat.probes of 1 means you are searching a single partition of your vector space. If the answer lives in a neighbouring list, it is not merely ranked low — it is never considered.

Measure it directly. Run your labelled queries with the index, then again with index scans disabled to get ground truth, and compare:

-- Ground truth: force an exact scan for this session.
-- Plain SET, not SET LOCAL — SET LOCAL only applies inside an explicit
-- transaction block, and outside one Postgres warns and discards it,
-- which would silently give you an index scan and a perfect-looking recall.
SET enable_indexscan = off;
SET enable_bitmapscan = off;
 
SELECT id, content, embedding <=> :query_vec AS distance
FROM documents
ORDER BY embedding <=> :query_vec
LIMIT 10;
 
-- Approximate: the path production actually takes
SET enable_indexscan = on;
SET enable_bitmapscan = on;
SET hnsw.ef_search = 40;   -- the default
 
SELECT id, content, embedding <=> :query_vec AS distance
FROM documents
ORDER BY embedding <=> :query_vec
LIMIT 10;

If the two lists differ meaningfully, raise ef_search until recall on your labelled set is acceptable, and treat the latency cost as the price of correctness. A retriever that answers in 20ms and misses the document is not fast — it is wrong, quickly.

The metric has to match the index

One quiet variant of this bug: an index built for one distance metric and queried with another. In pgvector, vector_cosine_ops and vector_l2_ops build different indexes, and using the mismatched operator in your ORDER BY means the planner won't select that index at all. You fall back to a sequential scan, which is slow but correct — so this one shows up as a latency problem long before anyone suspects retrieval. Check that the operator in the query matches the operator class in the index definition.

Cause 2: your chunk boundaries split the answer

If retrieval quality is uneven — great on some questions, hopeless on others — look at the chunks, not the index.

The classic failure is a fixed-size splitter cutting mid-explanation, so that the sentence defining a term lands in chunk 7 and the sentence applying it lands in chunk 8. Each chunk on its own is a weak match for the user's question. Neither ranks in the top k. Raising k doesn't help, because the problem is that no single chunk contains the answer.

Three fixes, in increasing order of effort:

• Overlap. A modest overlap between adjacent chunks (10–20% of chunk size) means boundary-spanning facts appear intact in at least one chunk. Cheap, and often enough.
• Split on structure, not character count. Markdown headings, HTML sections, function boundaries in code. A chunk that corresponds to a real semantic unit retrieves far better than one that corresponds to 512 characters.
• Small-to-big retrieval. Embed and search over small units for precision, but return the enclosing section to the model for context. You get the ranking behaviour of small chunks and the completeness of large ones.

A tell for this failure mode: your retrieved chunks are topically right but truncated — they discuss the correct subject and stop just before the useful part. That is a boundary problem, every time.

Cause 3: dense vectors are bad at exact tokens

Embeddings encode meaning. They are, by construction, not great at "find the document containing exactly ERR_CONN_4021" or getUserByEmail or a ticket number. A rare identifier contributes very little to a dense vector, so a semantically similar but incorrect document can easily outrank the exact match.

PresenceZero is a good illustration of the shape of corpus this bites hardest: almost all of its useful signal is exact — domain strings, RDAP status values, category names. Semantic similarity is simply the wrong tool for "does this exact token appear."

The fix is hybrid search: run a lexical query (BM25, or Postgres full-text search) alongside the vector query and merge the two ranked lists. The merge step is where people over-engineer. Reciprocal rank fusion avoids the tuning problem entirely — it scores by rank position rather than by raw relevance score, so you never have to normalise a BM25 score against a cosine distance:

def reciprocal_rank_fusion(result_lists, k=60):
    """Merge ranked ID lists. `k` is the rank constant; 60 is the common default."""
    scores = {}
    for results in result_lists:
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)
 
 
merged = reciprocal_rank_fusion([vector_results, bm25_results])

That is the whole algorithm — Elasticsearch implements the same 1 / (k + rank) formula with a rank_constant defaulting to 60, and its documentation makes the point plainly: "Not only does this remove the need to figure out what the appropriate weighting is using linear combination, but RRF is also shown to give improved relevance over either query individually."

Two practical notes. First, run both queries concurrently; done sequentially, hybrid search doubles your retrieval latency for no reason. Second, a document appearing in both lists gets both contributions, which is exactly the behaviour you want — agreement between two independent signals should outrank a strong showing in one.

Cause 4: you retrieved the right chunk and then buried it

If your log shows the correct passage in the retrieved set and the answer is still wrong, retrieval is done and the problem moved downstream.

The finding to know here is from Lost in the Middle: How Language Models Use Long Contexts (Liu et al.): performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle." The paper found this even in models built for long contexts.

The operational consequences are concrete:

• Retrieving more is not free. Going from k=5 to k=20 to be safe pushes your best chunk toward the middle of the prompt, where it is used least effectively. It can measurably reduce answer quality while improving retrieval recall — the two metrics move in opposite directions, which is why you must track both.
• Order deliberately. After ranking, put the highest-scoring chunks at the start and end of the retrieved block rather than in descending order. It is a few lines of code and costs nothing.
• Rerank instead of padding. A cross-encoder reranker over the top 30 candidates, keeping the best 5, is a better use of budget than passing 30 chunks to the model and hoping.

Long context windows changed the economics of this, but they did not repeal it. "It all fits" is not the same as "it is all used."

The order I actually work through it

1. Check the log. Is the right chunk being retrieved at all? This partitions the problem.
2. If not retrieved: raise the recall knob (hnsw.ef_search / ivfflat.probes) and re-measure. Cheapest possible fix, and often the whole bug.
3. Still not retrieved, and the near-misses look truncated? Fix chunking — overlap first, then structural splitting.
4. Still not retrieved, and the query hinges on an identifier? Add BM25 and fuse with RRF.
5. Retrieved but the answer is wrong? Reduce k, reorder so the best chunks sit at the edges, add a reranker.
6. Only now consider a different embedding model — and re-run the labelled set before and after, because a full re-embed is a real cost and you should be able to prove it bought something.

The theme across all of this is the same one that makes agent tooling work well: be deliberate about what enters the context window and in what order. I make the same argument about instruction files and MCP tool schemas in Claude Code vs Cursor — different surface, identical discipline.