7 Ways RAG Systems Fail in Production (And How to Test for Each)

Retrieval-Augmented Generation is the most widely deployed pattern in production GenAI applications. It is also the pattern with the most complex failure surface. RAG systems can fail at retrieval, generation, or the interaction between the two - and each failure mode requires different testing.

Here are the seven most common RAG failure modes we encounter in production assessments, with the specific testing approach and metrics for each.

1. Retrieval Miss

What happens: The correct document exists in the knowledge base, but the retrieval system does not return it. The model generates an answer without the information it needs.

Why it happens: Embedding quality issues, incorrect chunk boundaries, or retrieval parameters (top-k too low, similarity threshold too high) that exclude relevant documents.

How to test: Create a ground-truth dataset pairing queries with the documents that should be retrieved. Measure recall@k - the percentage of relevant documents that appear in the top-k retrieved results.

Metric: Context recall (RAGAS). Target: above 0.85.

2. Grounding Failure

What happens: The correct documents are retrieved, but the model generates claims that are not supported by the retrieved context. The model “adds” information from its parametric knowledge instead of staying grounded.

Why it happens: Insufficient grounding instructions in the system prompt. Context too long for the model to attend to fully. Or the query triggers parametric knowledge that overrides retrieved context.

How to test: Measure faithfulness - the proportion of generated claims that have explicit support in the retrieved context.

Metric: Faithfulness (RAGAS). Target: above 0.90.

3. Hallucination Despite Context

What happens: The model has the correct context available but generates an answer that contradicts it. The retrieved documents say one thing; the model says another.

Why it happens: This is a model-level failure where parametric knowledge conflicts with retrieved context. More common with questions where the model has strong prior beliefs from training data.

How to test: Design test cases where retrieved context contains information that contradicts common model knowledge. Verify the model follows the retrieved context, not its training data.

Metric: Custom contradiction detection rate.

4. Context Window Overflow

What happens: Too much context is retrieved and injected into the prompt, exceeding the model’s effective context window. The model loses track of relevant information in the noise.

Why it happens: Aggressive retrieval parameters (high top-k), large chunk sizes, or accumulation of conversation history that fills the context window.

How to test: Monitor context utilization and test answer quality at different context lengths. Identify the threshold where quality degrades.

Metric: Answer quality vs. context length correlation.

5. Stale Data

What happens: The knowledge base contains outdated information. The model generates answers based on obsolete data, presenting it as current.

Why it happens: Knowledge base update processes that lag behind source data changes. No staleness detection in the retrieval pipeline.

How to test: Include test queries with time-sensitive answers. Verify the application returns current information and handles staleness appropriately.

Metric: Temporal accuracy rate.

6. Contradiction Handling

What happens: Multiple retrieved documents contain contradictory information. The model picks one version without flagging the contradiction, or generates a confused synthesis.

Why it happens: Knowledge bases that contain multiple versions of the same information, conflicting sources, or documents from different time periods.

How to test: Seed the knowledge base with intentionally contradictory documents. Test whether the model detects and surfaces the contradiction or silently resolves it.

Metric: Contradiction detection rate.

7. Embedding Drift

What happens: RAG quality degrades over time as new documents are added with different vocabulary, formatting, or domain coverage than the original corpus.

Why it happens: Embedding models trained on the original corpus may not effectively represent new content. Semantic similarity thresholds calibrated for the initial knowledge base may not apply to an evolved one.

How to test: Track RAG evaluation metrics (faithfulness, context relevance, answer relevance) over time. Set up automated monitoring that alerts on quality degradation.

Metric: RAG metric trend over time (weekly measurement).

The Testing Framework

Test for all seven failure modes using a structured RAG evaluation pipeline:

1. Create a ground-truth dataset of 100+ query/document/answer triples
2. Measure retrieval metrics (recall, precision, context relevance)
3. Measure generation metrics (faithfulness, answer relevance, grounding rate)
4. Test edge cases (contradictions, staleness, overflow)
5. Establish baselines and monitor over time

The tools exist - RAGAS, Promptfoo, DeepEval - but the value is in test design and interpretation, not tool execution. Knowing which failure mode is causing your quality issues determines the fix.

Book a free scope call to discuss RAG evaluation for your specific pipeline.