Chunking methods for RAG

generated by nano

What is chunking?

To stay within a large language model’s token limit, we split text into chunks; a preprocessing step that breaks text into blocks. This lets the model process information without exceeding its context window.

What is RAG?

LLMs often suffer from hallucinations, generating false information with unearned confidence. This lack of factual “grounding” makes them unreliable for many high-stakes tasks.

To solve this, RAG (Retrieval-Augmented Generation) was introduced to provide LLMs with a “source of truth” to consult before answering.

To make RAG work, we first turn our documents into “digital fingerprints” called vector embeddings. We use specialized AI models (bi-encoders) to translate human text into these numbers, which are then stored in a vector database.

Think of it like a library: the quality of the search depends on how we filed the information. If our chunks of text are too big or too small, the AI won’t find the right answer. That’s why choosing a smart chunking strategy is just as important as the search method itself for getting accurate results.

Different Chunking Methods

1. Fixed-size chunking

This is the simplest approach. You pick a number of characters or tokens (e.g., 500 characters) and split the text at those intervals.

It’s fast and cheap, but it ignores the content. It often cuts sentences in half or separates a heading from its relevant paragraph, which can lose context during retrieval.

from langchain.text_splitter import CharacterTextSplitter

text = "Your long document text here..."
splitter = CharacterTextSplitter(
    separator="",
    chunk_size=500,
    chunk_overlap=50  # Overlap helps keep context between chunks
)
chunks = splitter.split_text(text)

2. Recursive chunking

A common default for general text. It walks down a hierarchy of separators. Double newlines, then single newlines, then spaces. If a paragraph is too big, it falls back to the next separator, aiming to keep related sentences together in a single block.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", " ", ""]
)
chunks = splitter.create_documents([text])

3. Document-specific chunking

3. Document-specific chunking

This method acknowledges that a Python script, an HTML page, and a Markdown file are structured differently. Instead of treating everything like a plain wall of text, it uses the document’s inherent formatting. HTML tags, # headers, and function definitions all set boundaries.

A single function, or a section of a manual, stays intact as a coherent unit. Best for structured data: Markdown, HTML, code.

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split_on = [
    ("#", "Header 1"),
    ("##", "Header 2"),
]

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
chunks = splitter.split_text(markdown_text)

4. Semantic chunking

4. Semantic chunking

This method looks at meaning. It reads the distance between sentences and keeps them together while they talk about the same topic. When the model detects a shift in subject matter, it breaks.

The result is chunks that vary in size but stay on topic. Requires an embedding model to judge whether two sentences belong together.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings

# It groups sentences by how similar they are in meaning
splitter = SemanticChunker(OpenAIEmbeddings())
chunks = splitter.create_documents([text])

5. Agentic Chunking

5. Agentic chunking

In this approach an LLM decides where to split. The agent reads the document and asks “does this part stand alone as a complete thought?”. It edits the document into logical pieces with high-level reasoning.

This is the most context-aware method, but also the slowest and the most expensive, because it needs multiple model calls just to prepare the data. In practice it’s a loop that asks an LLM whether each chunk is complete or needs more text.

# Conceptual Pseudo-code (usually implemented via LangGraph or custom loops)
def agentic_split(text):
    # 1. Start with a small piece of text
    # 2. Ask LLM: "Is this a complete thought?"
    # 3. If NO: Add next sentence and repeat.
    # 4. If YES: Create chunk and move to the next part.
    pass

Which one should you use?

What’s next?

That’s the high-level view of how we break down data for LLMs. Knowing the definitions is only half the work. In the coming weeks I’ll go through each strategy in detail, with the code, the common pitfalls, and the chunk-size settings that work.

Next up: A deeper look at semantic chunking for RAG.