Document preparation: from files to text a pipeline can read
Preparation decides what a retriever can find; extraction errors become missing answers.

Retrieval begins before retrieval. Whatever ends up in the index is whatever the preparation step produced, and nothing later in the pipeline can recover information that extraction discarded — a table flattened into a sentence, a two-column page read in the wrong order, a scanned contract whose text layer exists only as an image.
This sheet is about the stage that turns a corpus of files into text, structure and metadata. It is the least interesting part of a retrieval system and the one that most often explains a disappointing answer.
What the stage has to produce
Four things, and each has a consumer:
- text — what the embedding model and the generator will read;
- structure — headings, lists, tables, code blocks, and where each of them sits inside the document;
- metadata — at minimum the source and a location inside it (section, page, position), plus whatever the application needs in order to filter (see the chunking sheet);
- identity — a stable document identifier that survives a re-index, so that “we already have this” and “this replaced that” are answerable questions.
The first is unavoidable; the other three are decisions, and each of them is cheap now and expensive later.
Length limits are the first hard constraint
Chunking exists because the models have limits, and the documentation states them concretely: “the maximum length of input text for the Azure OpenAI text-embedding-3-small model is 8,191 tokens”. An embedding model has a context window of its own, and exceeding it does not raise an error in every pipeline — “exceeding this context window may means the excess tokens are truncated, or thrown away, before being processed into a vector”, which removes the tail of a document from the representation without telling anyone.
Preparation is therefore also where truncation is prevented: as the same vendor guidance puts it, partitioning content into chunks “helps you meet input token requirements and prevents data loss due to truncation”.
What each format costs you
PDFs. A PDF is a page description, not a document model. Text can be present as a text layer, or only as pixels in a scan (which is where OCR enters and where errors enter with it); multi-column layouts can be read in the wrong reading order; tables lose their grid when flattened; running headers and footers repeat on every page and, once chunked, appear as noise in dozens of chunks. Treating a PDF as “just text with a different extension” is the single most common preparation mistake.
Office documents, HTML and Markdown. These carry structure natively — headings, lists, tables — and the question is whether the extractor preserves it or flattens it. Structure is signal: a heading is often the best context a chunk will ever have.
Scanned and image material. Where there is no text layer, OCR is a measurement with an error rate rather than a conversion. Its mistakes become retrieval mistakes (the word you search for is a word the OCR misread), and the register of a pipeline should say which parts of the corpus went through it.
Preparation is where the chunk boundary is decided
The chunking sheet develops the choice; what belongs here is that preparation determines what a chunk can be. Anthropic’s description of the standard pipeline is “break down the knowledge base (the ‘corpus’ of documents) into smaller chunks of text, usually no more than a few hundred tokens” — a boundary that can only be respected if the extractor reported where sections, pages and paragraphs actually end.
Re-indexing is a first-class case
A corpus is not prepared once. Documents are added, corrected, superseded and deleted, and every one of those events is a preparation run with a decision attached:
- identity — the same document re-processed must replace its old entries rather than duplicate them;
- deletion — a withdrawn document has to leave the index as well as the folder, or the system will keep answering from a source that no longer exists;
- provenance — every chunk should still be able to name the file, version and location it came from, which is what makes a citation possible later.
What this stage is not
It is not the place to rewrite content for the model: summaries added to chunks are a deliberate technique with a cost and a measured effect, not a default (the chunking sheet treats it as one). And it is not the place to fix permissions: access control is applied when the index is queried, not when the text is extracted, and the access-control sheet in this layer describes why a copy loses the original’s permissions by default.
Level and prerequisites. L3 — integration level: the reader must be able to specify what preparation produces and to recognise the failure modes that come from the file formats rather than from the model. Prerequisites: the first sheet of this layer (whether retrieval belongs in the system at all).
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Applications — the node this sheet sits in.
References
- Microsoft — Chunk documents in Azure AI Search — the token limit of an embedding model, the truncation warning, the fixed-size and semantic chunking strategies, and the overlap recommendation.
- Pinecone — Chunking Strategies for LLM Applications — the definition of chunking, the two reasons it is necessary, and what exceeding an embedding model’s context window does to the data.
- Anthropic — Introducing Contextual Retrieval — the standard pipeline’s chunk size in the pipeline’s own words.