When a RAG answer is wrong, check two records before you touch chunk size. Was the correct evidence in the Top-k search results? Did it reach the final context the model actually received?
If it never made the Top-k, that’s a retrieval failure. If it was in the final context and the answer is still wrong, that’s a generation failure.
Comparing those two records can separate a good share of retrieval failures from generation failures. The cases below come from a RAG POC I ran on 200 to 300 internal technical and operations documents, with about 50 fixed questions.
For each question, I kept the search results, the final context, and the answer side by side. Answers that looked equally wrong had failed at different stages.
On this page
Two failures that look the same on screen
RAG (retrieval-augmented generation) is a setup where a search step finds documents and a model writes the answer from them. So a wrong answer can come from either half.
A retrieval failure is when the expected evidence document or sentence never makes it into the Top-k. A generation failure is when the evidence is in the final context, but the model misreads a condition or mixes in other content.
Evaluation tools split the same two axes. As of November 1, 2026, the Ragas documentation defines Context Recall as “how many of the relevant documents (or pieces of information) were successfully retrieved.”
It defines Faithfulness as “how factually consistent a response is with the retrieved context,” on a 0 to 1 scale. That’s a general standard. The cases and numbers below come from my own POC.
I didn’t start this way. At first, I assumed bigger chunks would cut broken context, so I grew chunks of around 500 tokens to around 1,000 and adjusted the overlap.
Even after I changed the settings, some questions kept failing. Chunk size alone didn’t explain them.
The five checks I run before changing chunking
My order now looks like this:
- Is the correct evidence in the source corpus?
- Did that document or chunk make it into the Top-k?
- Was the retrieved evidence included in the final context?
- Did the model use that evidence the way its conditions required?
- Does the answer’s citation point to the actual evidence?
Running these in order changes what I end up adjusting.
The stage where the evidence stops tells me what to look at next:
| Where the evidence stopped | Label | What I check |
|---|---|---|
| Not in Top-k | Retrieval failure | Retrieval |
| In Top-k, not in final context | Context construction error | Assembly rules |
| In final context, answer wrong | Generation failure | Generation |
| Source text or answer key wrong | Data/question error | Corpus, question set |
I also kept two more labels separate. A freshness error is when an older document took priority. A citation error is when the cited source was a different one.
The habit is close to debugging an agent that calls the wrong tool: the final answer alone doesn’t show which step broke, so you read the trace.
A retrieval failure from my POC: the overview doc won
One question asked about an actual operations procedure. The exact procedure lived in one specific operations document.
The search results ranked an overview document first instead. It shared many words with the question, and the final context didn’t contain the answer.
I checked the real ranks and scores in the captured log. Anonymized, the trace reads like this, keeping only what decides the failure: whether the evidence was included.
| Checkpoint | State in the log |
|---|---|
| Expected evidence | In a procedure section of an ops doc |
| Top-k | Expected chunk missing |
| Final context | Correct procedure not passed |
| Generated answer | Based on a similar overview doc |
| Error label | retrieval failure |
Read only the generated answer and it looks like the model made up a procedure. Step back one stage at a time, though, and the generation model never received the right material.
So the first thing to change here wasn’t the answer wording. It was the process that builds the search candidates.
If you only reword the prompt, the model still has a hard time using evidence it never got. The original RAG paper (Lewis et al., 2020) feeds retrieved passages into the generator, so missing passages mean missing material.
Still, I didn’t infer this case from the architecture. My logs showed the evidence never reached the final context.
For this question, I rechecked the query wording, the metadata, the retrieval method, and the document structure. I also ran some reranker comparisons.
Can a reranker fix a retrieval failure?
Only when the correct evidence is already among the first-stage candidates. A reranker reorders a list it’s given; it doesn’t add documents to that list.
The Amazon Bedrock reranker guide (checked November 1, 2026) lists the minimum input as a reranker model, the user query, and “a list of documents that the reranker must reorder.” If the right chunk isn’t in that list, reordering can’t create it.
Right chunk in the context, wrong answer anyway
The opposite case happened too. The correct chunk was in the Top-k and stayed in the final context.
But the model didn’t follow the condition in the evidence sentence as written. It merged in a condition from a different sentence, so I labeled it generation failure.
Retuning retrieval here would only touch a stage that had already worked. Both cases showed up as a wrong answer, but the first needed its search candidates fixed and the second needed its evidence-use rules checked.
For this one, I checked the grounded-answer conditions in the prompt. I also had the model return the evidence it used along with the answer.
Even a correct answer didn’t pass if its citation pointed to a different document. That’s why citation sits as its own check in the list above.
When the chunk was retrieved but never reached the prompt
That’s a separate problem from both failures. It’s the case where the correct chunk is retrieved, then dropped while the context is assembled.
I recorded it as a context construction error, a stage between retrieval and generation. Here the retrieval log looks fine; only the final context shows the gap.
When I passed many short snippets as context
In an earlier, separate POC, I passed Naver Search API results to an LLM as context (Naver is a Korean search engine).
When I added many short descriptions, snippets from different documents got mixed and the evidence became ambiguous. That POC isn’t part of the internal-corpus experiment above.
Putting in a lot of search results isn’t the same as delivering exactly the evidence the model needs.
Should you change chunk size first?
No. First confirm the evidence exists in the corpus and reached the Top-k and the final context. Then change the setting that matches the stage that failed.
I did compare three chunking setups on the same question set. The goal wasn’t to pick one winner. I wanted to see on which questions the evidence disappeared, and where a condition lost the heading it belonged to.
| Condition | What I had | Read with care |
|---|---|---|
| Corpus | ~200–300 tech/ops docs | Not a general benchmark |
| Questions | ~50 fixed | Type ratio not measured |
| Chunking | ~500 tokens, ~1,000 tokens, Markdown heading/section | No overlap values or hit rates |
| Logs | Query to answer, per question | No full benchmark harness |
The questions mixed short-answer, conditional, and multi-document types. The logs held the query, Top-k, document and chunk IDs, scores, the final context, and the answer.
Some questions clearly depended on keeping the document structure. When headings and section boundaries disappeared, it could become unclear which procedure a condition in the body belonged to.
That’s an observation from my POC. It doesn’t mean structure-preserving chunking wins for every RAG system.
Bigger chunks alone also didn’t always bring up an answer document that hadn’t been retrieved before. I’m not publishing exact overlap values, per-method hit rates, error counts, or improvement rates as confirmed statistics.
The POC was in Python. I used FAISS-based local experiments and a separate vector DB, and added a reranker in some runs.
I used several generation models. I tried to keep the question set and corpus snapshot fixed, but model versions, prompt hashes, and corpus commits weren’t all pinned. Don’t read it as a comparison of a specific model or chunking method.
What to log for each question
A RAG debugging log should keep the question, expected evidence, Top-k, final context, answer, and error label as one unit. That’s what lets you explain what to change next.
A minimal golden set (evaluation set) only needs the question, the expected answer, and the evidence document or section kept together. In my question set, I recorded the correct document and section where I could.
I looked more often at the rank the correct chunk landed at, and whether it reached the final context, than at the similarity score itself.
If I built the same experiment again, I’d bundle these for each question:
- query and expected answer
- evidence document ID and section
- retrieved document and chunk IDs, ranks, and scores
- results before and after the reranker
- the final context actually passed to the model
- generated answer and citation
- error label: retrieval, context construction, generation, citation, freshness, or data/question
Sharing traces from internal documents
For internal material, I generalize the questions and sentences and replace document IDs with something like DOC-001. Company names, system names, endpoints, internal document titles, and source text get masked.
Ranks, scores, and whether a chunk was in the Top-k and the final context can stay.
Chunking is an important variable, but it isn’t the name for every RAG error. My rule is still the same: check the Top-k and the final context before changing chunking.
Sources
- Ragas: Available metrics: RAG metrics including Context Recall and Faithfulness
- Ragas: Context Recall: definition, and why it needs a reference
- Ragas: Faithfulness: consistency of a response with the retrieved context, scored 0 to 1
- Amazon Bedrock reranker: a reranker reorders a given list of documents by relevance to the query
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: the original RAG paper (Lewis et al., 2020)
Related posts
- AI Agent Calling the Wrong Tool? I Cut It from 6/20 to 1/20
- Claude Outage Fallback: Request Failed, Files Still Changed
- HTTP 402 Payment Required: x402 Agent Spending Limits
- Claude Code Custom Commands to Skills: Why /project: Fails
I first wrote this in Korean on my blog: https://jjeongil.tistory.com/3559.