Day 114 · Tearing the book into useful pages

Chunking Strategies

You will be able to

Implement fixed-size, recursive, and structure-aware chunking on the same document
Explain the retrieval-precision vs context-completeness trade-off that chunk size controls
Choose overlap and size defaults for prose, tables, and code, and justify them
Attach metadata (source, section, position) at chunk time and say why it must happen there

Today's ~120 minutes

Spaced-rep: due cards + redraw the six RAG stages from memory (D113)	10 min
ELI5 + tech read; chunk-strategies visualizer	20 min
Guided: three tearings + the table rescue	40 min
Practice: chunk your own repo docs	20 min
Project: chunking decision card	20 min
Quiz + flashcards	10 min

Builds on: Day 113 — RAG architecture · Day 96 — Tokenization

The analogy

🧸 ELI5⚙ Tech

You're building the index for the open-book exam, and the book won't fit on index cards whole — you have to tear it into pages first. Tear carelessly and every page ends mid-sentence: the card that says "the limit is" is useless without the card that says "500 USD." Tear too small and each card is a confetti scrap with no context. Tear too big and each card is half the book — it matches every search vaguely and the student has to skim the whole thing anyway.

Chunking is the tearing strategy. The dumbest tear is "every 200 words, cut" (fixed-size). Smarter is "prefer to cut at paragraph breaks, then sentences, only then mid-sentence" (recursive). Smartest is "cut where the AUTHOR cut — at headings and sections — and write the chapter name on every scrap" (structure-aware). The strategy you pick decides what CAN ever be retrieved: retrieval returns chunks, not documents, so a fact split across two chunks is a fact your system half-knows forever.

Why this matters on the job

Chunking is the most underrated dial in RAG — practitioner postmortems routinely trace "the bot can't find things" to chunks that severed answers from their context, and no amount of embedding-model upgrades repairs a bad tear. It is also cheap leverage: re-chunking is a one-line config change plus re-index, versus weeks of model work. In your capstone (Day 119) and in any customer engagement, "show me your chunks for this failing query" is the first debugging move an experienced FDE makes — you are learning that move today.

Watch it happen

Tearing the book into useful pages — three chunkings, three fates

step 1 / 5

the document

§3 Employee leave: 16 weeks…

§3.2 Contractors: leave accrues

at 1 day per month worked,

up to 12 weeks maximum.

§4 Expense policy: receipts…

One HR document, one question coming later: "How much parental leave do contractors get?" How we tear the doc into chunks decides whether retrieval can ever answer it.

🎯 Predict modespeed1×

Guided practice

guided 1

One document, three tearings

25 min

Create chunk_lab.py with the starter code: one realistic markdown policy doc and three chunkers — fixed (with overlap), recursive, structure-aware.
Run it. For each strategy, read the printed chunks and count: how many chunks cut a sentence in half? How many separated a number from the rule it belongs to?
Now run the retrieval comparison at the bottom: the same three queries against each strategy's index (reusing Day 113's cosine retriever).
Query "what is the travel booking limit" should expose the fixed chunker: the phrase "500 USD" and the words "travel" or "pre-approval" land in different chunks. Verify which strategies return a chunk containing the COMPLETE answer.
Note how the structure-aware chunks carry their heading path in the text — and how that alone lifts their scores on section-flavored queries.

🐍 python — editable, runs in your browser

import math, re
from collections import Counter

DOC = """# Nimbus Analytics — Expense & Travel Policy

## Purchases
Individual purchases over 500 USD require written manager pre-approval.
Software subscriptions of any amount must go through IT procurement.
Receipts are mandatory for every expense above 25 USD.

## Travel
All travel must be booked through the Navan portal. Economy class is
standard for flights under 6 hours. Hotel budget is capped at 250 USD
per night in tier-1 cities and 180 USD elsewhere.

## Reimbursement
Submit expense reports within 30 days of purchase. Reimbursement lands
in the next payroll cycle after approval. Late reports need VP sign-off.
"""

# --- strategy 1: fixed-size with overlap ---
def chunk_fixed(text, size=180, overlap=40):
    text = re.sub(r"\s+", " ", text).strip()
    chunks, start = [], 0
    while start < len(text):
        chunks.append(text[start:start + size])
        start += size - overlap
    return chunks

# --- strategy 2: recursive (paragraphs, then sentences) ---
def chunk_recursive(text, size=180):
    def split(piece, seps):
        if len(piece) <= size or not seps:
            return [piece]
        parts, out, buf = piece.split(seps[0]), [], ""
        for p in parts:
            cand = (buf + seps[0] + p) if buf else p
            if len(cand) <= size:
                buf = cand
            else:
                if buf: out.append(buf)
                out.extend(split(p, seps[1:]) if len(p) > size else [p])
                buf = ""
        if buf: out.append(buf)
        return out
    return [c.strip() for c in split(text, ["\n\n", ". ", " "]) if c.strip()]

# --- strategy 3: structure-aware (markdown headings + heading-path metadata) ---
def chunk_structured(text):
    chunks, h1, h2, buf = [], "", "", []
    def flush():
        if buf:
            path = " > ".join(p for p in (h1, h2) if p)
            body = " ".join(buf)
            chunks.append({"path": path, "text": path + ": " + body})
            buf.clear()
    for line in text.splitlines():
        if line.startswith("## "):
            flush(); h2 = line[3:].strip()
        elif line.startswith("# "):
            flush(); h1 = line[2:].strip(); h2 = ""
        elif line.strip():
            buf.append(line.strip())
    flush()
    return chunks

# --- retrieval comparison (Day 113's retriever) ---
def embed(t): return Counter(re.findall(r"[a-z0-9']+", t.lower()))
def cosine(a, b):
    dot = sum(a[w] * b[w] for w in a if w in b)
    na = math.sqrt(sum(v * v for v in a.values()))
    nb = math.sqrt(sum(v * v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0

def top1(query, chunks):
    qv = embed(query)
    return max(chunks, key=lambda c: cosine(qv, embed(c)))

STRATS = {
    "fixed": chunk_fixed(DOC),
    "recursive": chunk_recursive(DOC),
    "structured": [c["text"] for c in chunk_structured(DOC)],
}
for name, chunks in STRATS.items():
    print(f"=== {name}: {len(chunks)} chunks ===")
    for c in chunks: print("  |", c[:80].replace("\n", " "))

for q in ["what is the travel booking limit",
          "hotel budget per night",
          "when do I get reimbursed"]:
    print("\nQ:", q)
    for name, chunks in STRATS.items():
        print(f"  {name:10s} ->", top1(q, chunks)[:90])

Ctrl/⌘+Enter runs · Tab indents · numpy/pandas/sklearn auto-load on import (torch and network calls need a local run)

guided 2

Break a table, then save it

15 min

Add this markdown table to the doc, under a new "## Per-diem rates" section: Region | Breakfast | Dinner — three data rows (US 20/50, EU 15/45, APAC 12/40, as pipe-separated lines).
Re-run the fixed chunker and find where the table got cut. Ask: if retrieval returns only the chunk with the "APAC" row, can a model know 12 means breakfast USD? (No — the header is in another chunk.)
Implement the fix: detect table lines (they contain " | ") and emit the table as ONE atomic chunk, header included. If a table exceeds the size limit, split by row-groups but prepend the header row to every group.
Re-run retrieval for "APAC dinner per diem" and confirm the atomic-table strategy returns a self-sufficient chunk.

On your own

Chunk your own repo docs

20 min

Take a real README or docs file from one of your own projects (Day 21 or Day 42 work). Run all three chunkers on it. Deliverable: a 10-line written verdict — which strategy wins for THIS document and why; the worst single chunk each strategy produced (paste it); and your recommended size/overlap numbers with one sentence of justification each.

Constraints: judge chunks by asking "could a model answer a question from this chunk alone?" Hints: code fences are the tables of READMEs — check whether any chunker cut one in half; if your doc has headings, structure-aware should win, so explain any surprise.

Ship before you stop

The chunking decision card

Create chunking_notes.md: a decision card you will reuse at capstone ingest time (Day 119). Contents: (1) a table of the three families with one-line mechanism, best-for, and failure mode; (2) your defaults — size in tokens, overlap %, separator priority — for prose, markdown docs, tables, and code, each with a one-sentence why; (3) the "torn answer" example from today's lab pasted verbatim as a cautionary exhibit; (4) three rules that must happen at chunk time (metadata stamping, table atomicity, heading-path prefixing). Commit it.

Rubric — check what you completed (0/5)

All three families compared with mechanism, best-for, and failure modeConcrete defaults per content type (prose/markdown/tables/code) in tokens, each justifiedA real severed-answer chunk from the lab included as evidenceThe three chunk-time rules stated with reasonsCommitted to the practice repo

Common mistakes & misconceptions

Chunking by characters while budgeting prompts by tokens. Measure chunk size in tokens (Day 96) — a 1000-character code chunk can be 400 tokens; emoji-heavy text even more.
Assuming bigger chunks are safer. Big chunks blur the embedding (many topics → one vector), tanking retrieval precision. Retrieve-small, deliver-big is the real fix (Day 117).
Using overlap as a substitute for good boundaries. Overlap patches cut wounds; recursive/structure-aware splitting avoids making them.
Splitting tables from their headers or code mid-function. Keep them atomic, or replicate the header into every fragment.
Deferring metadata to "later." After chunking, you cannot recover which section a chunk came from — stamp source, heading path, and position at chunk time.
Re-chunking without re-embedding and re-indexing everything. Chunks and their vectors must stay in lockstep; a partial re-index silently mixes two chunking generations.

Knowledge check

Q1. Retrieval keeps returning a chunk that reads "...requires written manager" — the amount is in the previous chunk. What is the cleanest fix?

Q2. What is the core trade-off that chunk size controls?

Q3. Why must metadata be attached at chunk time rather than later?

Go deeper — curated resources

articlePinecone Learning Center — chunking strategies ↗20 min docsLangChain docs — text splitters ↗15 min docsLlamaIndex — node parsing & ingestion ↗15 min

If you have a third hour

Semantic chunking — Embed sentences, cut where adjacent-sentence cosine similarity drops below a threshold. Elegant, ~10–50x the embedding cost at ingest; try it when structure-aware boundaries still feel wrong.

Done means

All three chunkers run; severed-answer failure demonstrated and explained
Table made atomic and verified retrievable as a self-sufficient chunk
Own-docs verdict written with recommended defaults
Decision card committed
Quiz ≥ 2/3

How this connects

← Back: Day 113 showed chunking as one of six stages — today you learned it decides what can ever be retrieved. Day 96's tokenization is why sizes are measured in tokens, not characters.

Forward →: Day 115 embeds these chunks for real; Day 117's parent-child retrieval resolves today's size trade-off outright. On Day 119 your capstone ingest pipeline applies today's decision card, and Day 136 measures chunking quality with context-precision metrics.

Unlocks: D116 Hybrid Search