Agentic Chunking - Why Your RAG Pipeline Is Quietly Failing (And How to Fix It)

TL;DR: Bad chunks return broken context, and the model fills the gaps. Agentic chunking makes an LLM reason about meaning, not mechanics.

Cover

generated by nano

The real reason RAG fails

You embedded your docs. You picked a great model. Answers are still shallow and wrong.

The culprit isn’t your model. It’s your chunks.

Split ideas in the wrong place and your retriever returns broken context. Your model fills the gaps with a guess. Traditional chunking optimizes for speed. Agentic chunking optimizes for understanding.

The Chunking Showdown

Method	How It Works	Strengths	Weaknesses
Fixed-Size	Split every N tokens	Fast, cheap	Cuts mid-idea, mixes unrelated concepts
Semantic	Split at similarity boundaries	More natural breaks	Still static, misses long-range links
Proposition-Based	Extract atomic facts first	High granularity, factually precise	Can feel fragmented without smart grouping
Agentic	LLM decides chunk membership + evolves metadata	Meaning-first, dynamic, coherent	More LLM calls, higher indexing cost

What makes agentic chunking different

It works like an editor, not a pair of scissors:

Generalizes across vocabulary. apples, pizza, and sushi all become food_preferences
Evolves chunk metadata. Titles and summaries refresh as new content is added, improving retrieval ranking
Handles real content. Blogs, research notes, and docs that repeat or shift topics don’t break it

The core loop

Built on the Dense X Retrieval paper (arXiv:2312.06648) which proved propositions outperform sentences and passages as retrieval units:

Dense X Retrieval Figure

generated by nano

for proposition in extract_propositions(document):
    chunk_id = find_relevant_chunk(proposition, chunk_outline)
    if chunk_id:
        add_to_chunk(chunk_id, proposition)
        refresh_metadata(chunk_id)
    else:
        create_new_chunk(proposition)

For reliable structured output from the LLM:

class ChunkID(BaseModel):
    chunk_id: str | None = None

Want to try this in LangChain? Here is the community prompt: kumja/proposal-indexing on LangSmith Hub

The corn test

Your knowledge base contains three corn-related facts. Fresh corn, corn tortillas, and high-fructose corn syrup.

A fixed-size chunker mashes them together. A query about healthy snacks retrieves the corn syrup fact too, polluting the context.

An agentic chunker separates them into fresh_produce, traditional_cuisine, and food_additives. The right chunk, the right query, the right answer.

If your chunking can handle corn, it can handle production data.

Honest tradeoff

Need ultra-low cost and latency? Semantic chunking is fine.

Need retrieval quality you can trust? Agentic chunking is worth the extra LLM calls. It pays off most for long-form docs, overlapping topics, or any pipeline where a wrong answer has real consequences.

FAQs

Is agentic chunking the same as proposition-based chunking? No. Propositions are the raw material. Agentic chunking is the assembly process that groups them intelligently.

Does it work with any embedding model? Yes. It’s a pre-processing step. Embed with whatever you prefer after chunking.

Best model to use as the agent? Smaller models (Haiku, GPT-4o-mini) handle chunk assignment well. Use larger models for generating high-quality titles and summaries.

Good for real-time ingestion? Best for batch indexing. For real-time, run semantic chunking fast and agentic chunking async in the background.

Research: Chen et al. (2023). Dense X Retrieval. arXiv:2312.06648 LangSmith prompt: kumja/proposal-indexing