RAG Architecture, Failure Modes, and Evaluation

Retrieval-augmented generation (RAG) combines a generative model with information retrieved at request time. The original RAG research paired a sequence-to-sequence model's parametric memory with a dense index used as non-parametric memory. Production systems now use many retrieval patterns, including lexical, dense, hybrid, structured, graph, and tool-based retrieval.

RAG can make knowledge easier to update, constrain, inspect, and cite. It does not make an answer true by construction. Reliability depends on the source corpus, eligibility rules, retrieval, prompt construction, generator behavior, citation verification, and operating controls.

A production RAG system is a pipeline

1. Ingest and govern sources. Record owner, origin, rights, version, effective dates, sensitivity, authorization policy, and deletion requirements.
2. Parse and index. Preserve document structure and identifiers; choose chunking, metadata, embeddings or lexical fields, and update behavior appropriate to the content.
3. Authorize before retrieval. Apply tenant, user, purpose, and document-level policy so ineligible content cannot enter the candidate set.
4. Retrieve and rerank. Generate candidates, filter them, and rank evidence for the request. Structured queries or deterministic APIs may be better than semantic search for exact facts.
5. Construct the model context. Separate instructions from untrusted retrieved content, retain source identifiers, manage token limits, and represent conflicts or missing evidence.
6. Generate with evidence rules. Require supported answers, source-linked citations, uncertainty where appropriate, and abstention when eligible evidence is inadequate.
7. Verify and monitor. Check citation entailment and policy compliance; log versions and retrieval traces subject to privacy controls; monitor failures, drift, incidents, latency, and cost.

Choose retrieval for the evidence shape

Common retrieval modes and their trade-offs

Mode	Useful when	Primary risk to test
Lexical search	Exact terms, codes, names, citations, or uncommon vocabulary matter	Synonyms and paraphrases may be missed
Dense retrieval	Semantic similarity and paraphrase matching matter	Similar is not necessarily relevant or authoritative
Hybrid retrieval	Both exact and semantic matching are valuable	Fusion and reranking can hide poor component behavior
Structured query	The answer is in a governed database with typed fields	Incorrect query generation, joins, units, or row-level access
Graph retrieval	Relationships, paths, and entity identity drive the task	Entity resolution and incomplete or stale edges
Tool or API retrieval	Fresh operational state comes from an authoritative service	Tool permission, validation, side effects, availability, and freshness

Start with the simplest method that satisfies the task. A vector database is not a mandatory RAG component.

Failure modes occur before and after generation

Failure modes and observable controls

Layer	Example failure	Evidence or control
Corpus	Missing, obsolete, duplicated, contradictory, low-quality, or unlicensed source	Source register, owner, version/effective date, quality review, conflict policy, and deletion propagation
Authorization	A user retrieves material they are not allowed to see	Pre-retrieval policy enforcement, negative permission tests, tenant isolation, and access audits
Parsing/chunking	A qualification, table header, footnote, or section relationship is separated from the claim	Structure-aware parsing, representative retrieval tests, and stable source anchors
Retrieval	Relevant evidence is absent or outranked by a superficially similar passage	Recall at k, ranking metrics, hard negatives, query slices, and retrieval error analysis
Prompt/context	Retrieved text contains an instruction or malicious payload that competes with system policy	Treat sources as untrusted data, isolate instructions, restrict tools, test indirect prompt injection, and validate outputs/actions
Generation	The answer contradicts or goes beyond retrieved evidence	Claim-level citation support, faithfulness tests, abstention policy, and calibrated human review
Operations	An index, model, prompt, embedding, reranker, or source changes silently	Versioned lineage, release gates, rollback, monitoring, and re-evaluation triggers

Evaluate components and the complete task

Build a test set from the intended domain, users, permissions, source types, and costly failure modes. Include answerable, unanswerable, ambiguous, conflicting, multilingual, stale, permission-denied, and adversarial cases. Keep a held-out set and version it with the corpus.

Minimum evaluation layers

Layer	Questions	Example measures
Retrieval	Did the system return eligible, authoritative evidence and rank it usefully?	Recall@k, precision@k, MRR or nDCG where labels support them, coverage by slice, and authorization violations
Generation	Does the answer address the request and stay within the retrieved evidence?	Answer relevance, claim-level faithfulness, citation precision/coverage, correctness against a reference where available, and unsupported-claim rate
Abstention	Does the system decline when evidence is absent, conflicting, stale, or forbidden?	False-answer and false-abstention rates under defined thresholds
End-to-end task	Does the system improve the real workflow without shifting unacceptable work or risk?	Task success, time, downstream error, human correction, escalation, user comprehension, and outcome by relevant slice
Operations	Can the service meet its performance and control objectives?	Latency percentiles, throughput, availability, per-task cost, index freshness, incident rate, and recovery time
Security/privacy	Can content, prompts, tools, or permissions be abused?	Prompt-injection success, data disclosure, cross-tenant retrieval, tool misuse, logging exposure, and deletion tests

Frameworks such as RAGAS and ARES offer automated signals for context relevance, answer faithfulness, and answer relevance. They do not replace target-domain ground truth or human review. Validate any model-based judge against qualified human labels and report uncertainty.

Use citations as verifiable links, not decoration

A citation should identify the exact eligible source passage supporting a claim. Test whether the cited passage actually entails the claim, whether important claims have citations, and whether the answer accurately represents qualifications, dates, and disagreement. A source link beside an unsupported statement is not grounding.

RAG does not have an inherent cost advantage

A RAG system adds ingestion, storage, indexing, retrieval, reranking, context tokens, authorization, evaluation, and operations. It may reduce update latency, avoid some fine-tuning, or allow a smaller generator for a defined task. Measure the net result against a relevant baseline.

• Report cost per successful task, not only token or query price.
• Include indexing, embeddings, storage, network, model calls, retries, observability, human review, and engineering.
• Measure quality, latency, throughput, availability, and cost together.

High-stakes domains need stronger release gates

For healthcare, legal, finance, education, public services, critical infrastructure, or cybersecurity, do not convert retrieved text directly into consequential action. Define authoritative sources and jurisdiction, require domain review, preserve source dates and versions, test affected groups and accessibility, provide a fallback, and give a qualified human the information, time, authority, and interface needed to intervene.

RAG does not establish that advice is clinically valid, legally current, compliant, secure, or suitable for a specific person. The system should clearly communicate scope, uncertainty, source status, and escalation paths.

A practical release checklist

• Every source has an owner, provenance, rights, sensitivity, effective date, and deletion/update path.
• Authorization is enforced before retrieval and tested with negative cases.
• Retrieval and generation are evaluated separately and end to end on representative data.
• Answers expose stable citations and abstain under documented evidence conditions.
• Indirect prompt injection, corpus poisoning, sensitive-data leakage, and tool abuse are tested.
• Models, prompts, parsers, indexes, embeddings, rerankers, and corpora are versioned.
• Production changes trigger re-evaluation; incidents have owners, rollback, and corrective-action records.

Conclusion

RAG is an architecture for connecting generation to external evidence. Its advantage is controllability and testability—not guaranteed truth. A credible implementation makes source eligibility, retrieval quality, answer faithfulness, permissions, abstention, security, and operating cost observable.

Next in the AI Innovation Series: AI Data Centers—Capacity, Energy, and Operational Trade-offs →