RAG Chunking Strategies That Actually Work

The Problem#

Bad chunking breaks RAG quietly. The retriever may return a paragraph that matches the query but excludes the requirement, exception, warning, or code sample needed to answer correctly.

Why It Matters#

Chunking controls the unit of retrieval. If chunks are too small, context is fragmented. If chunks are too large, retrieval becomes noisy and expensive.

Core Concepts#

Chunk by meaning before chunking by token count. Headings, sections, tables, code blocks, and FAQ entries are natural boundaries. Metadata such as document version, owner, product, language, and effective date is part of retrieval quality.

Implementation#

Use a simple policy first:

split by heading
keep code blocks intact
target 500-900 tokens
allow 10-15 percent overlap
store parent document id and heading path
attach version and access metadata

For long technical docs, retrieve child chunks but include a small parent summary or neighboring section when generating the final answer.

Example#

Suppose a deployment runbook has this structure:

# Deploying the Billing Service
## Pre-checks
## Deploy command
## Rollback command
## Known failure: schema lock timeout

A bad splitter may cut the "Known failure" section into two chunks: one with the error message and another with the fix. The retriever returns only the error message, so the model cannot explain the rollback.

A better chunk keeps the failure section together:

{
  "text": "Known failure: schema lock timeout... Fix: stop migration job, rollback version 42, then retry after lock clears.",
  "metadata": {
    "documentId": "billing-deploy-runbook",
    "headingPath": ["Deploying the Billing Service", "Known failure: schema lock timeout"],
    "service": "billing",
    "version": "2026-08"
  }
}

Now the query "billing deploy stuck on schema lock" retrieves both the symptom and the fix in one useful chunk.

Common Mistakes#

• Splitting blindly every 1,000 characters.
• Breaking tables or code examples across chunks.
• Omitting metadata needed for filtering.
• Changing chunking without re-embedding and re-evaluating.

Production Considerations#

Different content types need different chunking. Runbooks, API docs, support tickets, design docs, and legal policies should not share one universal splitter.

Security#

Preserve access-control metadata at the chunk level. A public paragraph inside a private document may still reveal sensitive context through neighboring text.

Performance#

Smaller chunks improve precision but increase index size and candidate count. Measure both retrieval quality and latency before standardizing a chunk size.

Summary#

Good RAG chunking respects document structure, keeps useful context together, and stores the metadata needed to retrieve safely and accurately.