If I had a dollar $ for every time someone explained RAG in exactly four boxes and an arrow between each, I’d have enough to fine-tune a small LLM by now.
Here’s the thing. Those four boxes aren’t wrong. They’re just the skeleton. And a skeleton without organs, blood flow, and a nervous system doesn’t walk anywhere. It just lies there looking like it should work.
So before you nod along to the “it’s simple” version, sit with these for a second:
- Did your parser actually capture the table on page 14, or did it turn into word soup?
- That chart your document had. Does your pipeline even know it existed?
- Why that chunk size? Why that overlap? Did you pick it, or did a tutorial pick it for you?
- Your vector DB choice. Was that a real decision, or the first result on Google?
- The 5 chunks you retrieved. Are they relevant, or just similar-sounding?
- Is there noise riding along with the signal, diluting your answer?
- How do you know the LLM’s answer is actually grounded in what you retrieved, and not just plausible?
That’s the difference between a RAG demo that works once and a RAG system that handles real documents correctly.
The real flow (bird’s-eye view)
Think of it as a pipeline where each step checks the work of the last one:
| Stage | What’s actually happening | The question nobody asks |
|---|---|---|
| Parsing | Documents → clean structured text | Did tables/images survive, or vanish? |
| Chunking | Splitting text into digestible pieces | Why this size? Why this overlap? |
| Embedding | Turning chunks into vectors | Does this model “get” your domain? |
| Storage | Vectors land in a DB | Picked for hype, or for your scale/latency needs? |
| Hybrid Search | Keyword (BM25) + semantic search | Are you only doing vector search and missing exact matches? |
| Metadata Filtering | Narrowing by source/date/dept | Or is everything just dumped into one giant pile? |
| Reranking | Cross-encoder re-scores top candidates | Or are you trusting raw similarity scores blindly? |
| Context Selection | Picking the final Top-K chunks | Too few = missing info. Too many = confused LLM. |
| Generation | LLM writes the answer | Grounded in your docs, or politely hallucinating? |
| Answer Relevancy | Did it actually answer the question | Anyone checking, or just shipping it? |
Every row above has its own failure modes and trade-offs, each of which is worth a post on its own.
Why this matters
A “simple” RAG pipeline fails silently. It doesn’t crash. It gives you a confidently wrong answer, citing a chunk that’s 70% irrelevant, built from a table your parser butchered, retrieved because it was vector-similar rather than actually-useful. No one notices until a user does.
Good RAG isn’t about stacking the four boxes. It’s about making each step accountable: parsing for fidelity, chunking for context, retrieval for relevance, generation for grounding.
What’s next
That was a high-level view, intentionally shallow. Next I’ll work through each stage, starting with document parsing, which is where most RAG pipelines actually break.
Stay tuned.
Inspired by my own hurdles.