Articles

Chunking and metadata: the shape of what gets retrieved

A chunk is the unit of retrieval, of context and of citation; its size decides all three.

Reading: 6 minAutomation & AI

Article cover: Chunking and metadata: the shape of what gets retrieved

A chunk is not an implementation detail. It is the unit that gets embedded, the unit the retriever returns, the unit the generator reads, and — if the interface shows sources — the unit you are able to cite. Choosing its size and its boundaries decides three things at once, and the choice is made at index time, which means changing it later means re-indexing the corpus.

Two reasons to chunk, and one trade-off

Chunking is “the process of breaking down large text into smaller segments called chunks”, an “essential preprocessing technique that helps optimize the relevance of the content ultimately stored in a vector database”. The documentation gives two reasons, and they pull in opposite directions: “to ensure embedding models can fit the data into their context windows, and to ensure the chunks themselves contain the information necessary for search”.

The trade-off is stated just as plainly: “The trick lies in finding chunks that are big enough to contain meaningful information, while small enough to enable performant applications and low latency responses for workloads such as retrieval augmented generation and agentic workflows.” Size is therefore not a setting to copy from a blog post; it is a balance between what a chunk can mean on its own and what a query can afford to carry.

The two families

Fixed size. Split by a fixed number of characters or tokens, with an overlap. The vendor guidance offers a starting point rather than a rule: “Define a fixed size that’s sufficient for semantically meaningful paragraphs (for example, 200 words or 600 characters) and allows for some overlap (for example, 10-15% of the content)”, and, elsewhere on the same page, “We recommend starting with a chunk size of 512 tokens (approximately 2,000 characters) and an initial overlap of 25%, which equals 128 tokens.”

Overlap exists for a reason that is easy to forget: a boundary cut through a sentence leaves two chunks that are each missing half of an idea, and the overlap restores continuity — “overlapping a small amount of text between chunks can help maintain continuity and context”, while “the optimal overlap might vary depending on your content type and use case”.

Structure-aware. Split where the document itself changes subject: “Break content into meaningful units that preserve context and semantic relationships across sentences and paragraphs. This approach produces chunks that better maintain semantic coherence and can span page boundaries.” The same documentation notes a page-based variant (“splitting by pages (defined by character length)”) and a hard constraint: this family only works if preparation preserved the boundaries (see the previous sheet).

Neither family is right in general. Fixed size is predictable and blind; structure-aware is faithful and depends on the extractor.

The third option: add context to the chunk

A chunk that says “the maximum is 200 units” is unambiguous only next to the sentence that came before it, and retrieval sees the chunk alone. The response to this is a preprocessing technique that prepends a short description of the surrounding document to every chunk before embedding it, so that the fragment’s context travels with it — the technique Anthropic calls contextual retrieval, whose measured effect on retrieval failures is registered in the retrieval-failures sheet of this layer.

Two properties matter when it is proposed for a project: it is applied at indexing time to every chunk (a cost that is paid once per document, and re-paid on re-index), and the note in the source material is that it is a technique to be validated — “always run evals” — not a default.

Metadata is what makes a chunk addressable

A vector is not addressable. What makes a chunk usable by the rest of the system is the payload attached to it, and the stores treat that payload as a first-class, filterable object: “For performant filtering, create payload indexes for the fields you plan to filter on. For best results, create payload indexes before ingesting data.”

The fields a chunk needs, and who consumes each:

Field Consumer
source document identity and version citation, deletion, de-duplication
position (section, page, offset) citation, “show me the context around this”
permission or tenancy tags the access filter applied at query time
document or chunk date freshness rules, conflict resolution
structural type (heading, table, code, prose) reranking and context assembly

In a relational store the same thing is a column, which is why the vector extension’s own documentation shows the pattern of filtering and ordering in one statement — and notes that the ordering must be combined with a limit for the index to be used at all.

One failure to avoid by design: a chunk whose metadata cannot be filtered on is a chunk that will be returned to the wrong reader or never returned at all. Adding the field later means re-indexing, for the same reason as changing the size.

What size does to the rest of the pipeline

  • Too small — precise matching, but each hit carries little context; the generator receives fragments, and the citation points at something that is meaningless out of context.
  • Too large — good context per hit, fewer hits, more tokens per query, and a higher chance that the answer is diluted inside the chunk; the position effect described in the retrieval-failures sheet applies to whatever is put in front of the model.
  • Overlap too generous — the same text arrives twice, consuming the top-k budget and producing the appearance of two sources where there is one.
  • Inconsistent — a corpus chunked by three different rules at three different times is a corpus whose retrieval quality cannot be reasoned about.

Level and prerequisites. L3 — integration level: the reader must be able to choose a chunking strategy for a given corpus and to specify the metadata a chunk must carry. No chunker is configured and no index is measured here. Prerequisites: the previous sheet of this layer (what preparation can preserve).

Where to go next

References

  • Pinecone — Chunking Strategies for LLM Applications — the definition of chunking, the two reasons for it and the size trade-off.
  • Microsoft — Chunk documents in Azure AI Search — the two families, the overlap guidance, the 512-token starting point and the structure-aware alternative.
  • Qdrant — Filtering — payload indexes and when to create them.
  • Anthropic — Introducing Contextual Retrieval — the technique of adding context to a chunk, and the instruction to evaluate rather than assume.
  • pgvector — README — the relational form of the same design: metadata as columns, filter and ordering in one statement.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.