Articles

Document preparation: from files to text a pipeline can read

Preparation decides what a retriever can find; extraction errors become missing answers.

Reading: 6 minAutomation & AI

Article cover: Document preparation: from files to text a pipeline can read

Retrieval begins before retrieval. Whatever ends up in the index is whatever the preparation step produced, and nothing later in the pipeline can recover information that extraction discarded — a table flattened into a sentence, a two-column page read in the wrong order, a scanned contract whose text layer exists only as an image.

This sheet is about the stage that turns a corpus of files into text, structure and metadata. It is the least interesting part of a retrieval system and the one that most often explains a disappointing answer.

What the stage has to produce

Four things, and each has a consumer:

  • text — what the embedding model and the generator will read;
  • structure — headings, lists, tables, code blocks, and where each of them sits inside the document;
  • metadata — at minimum the source and a location inside it (section, page, position), plus whatever the application needs in order to filter (see the chunking sheet);
  • identity — a stable document identifier that survives a re-index, so that “we already have this” and “this replaced that” are answerable questions.

The first is unavoidable; the other three are decisions, and each of them is cheap now and expensive later.

Length limits are the first hard constraint

Chunking exists because the models have limits, and the documentation states them concretely: “the maximum length of input text for the Azure OpenAI text-embedding-3-small model is 8,191 tokens”. An embedding model has a context window of its own, and exceeding it does not raise an error in every pipeline — “exceeding this context window may means the excess tokens are truncated, or thrown away, before being processed into a vector”, which removes the tail of a document from the representation without telling anyone.

Preparation is therefore also where truncation is prevented: as the same vendor guidance puts it, partitioning content into chunks “helps you meet input token requirements and prevents data loss due to truncation”.

What each format costs you

PDFs. A PDF is a page description, not a document model. Text can be present as a text layer, or only as pixels in a scan (which is where OCR enters and where errors enter with it); multi-column layouts can be read in the wrong reading order; tables lose their grid when flattened; running headers and footers repeat on every page and, once chunked, appear as noise in dozens of chunks. Treating a PDF as “just text with a different extension” is the single most common preparation mistake.

Office documents, HTML and Markdown. These carry structure natively — headings, lists, tables — and the question is whether the extractor preserves it or flattens it. Structure is signal: a heading is often the best context a chunk will ever have.

Scanned and image material. Where there is no text layer, OCR is a measurement with an error rate rather than a conversion. Its mistakes become retrieval mistakes (the word you search for is a word the OCR misread), and the register of a pipeline should say which parts of the corpus went through it.

Preparation is where the chunk boundary is decided

The chunking sheet develops the choice; what belongs here is that preparation determines what a chunk can be. Anthropic’s description of the standard pipeline is “break down the knowledge base (the ‘corpus’ of documents) into smaller chunks of text, usually no more than a few hundred tokens” — a boundary that can only be respected if the extractor reported where sections, pages and paragraphs actually end.

Re-indexing is a first-class case

A corpus is not prepared once. Documents are added, corrected, superseded and deleted, and every one of those events is a preparation run with a decision attached:

  • identity — the same document re-processed must replace its old entries rather than duplicate them;
  • deletion — a withdrawn document has to leave the index as well as the folder, or the system will keep answering from a source that no longer exists;
  • provenance — every chunk should still be able to name the file, version and location it came from, which is what makes a citation possible later.

What this stage is not

It is not the place to rewrite content for the model: summaries added to chunks are a deliberate technique with a cost and a measured effect, not a default (the chunking sheet treats it as one). And it is not the place to fix permissions: access control is applied when the index is queried, not when the text is extracted, and the access-control sheet in this layer describes why a copy loses the original’s permissions by default.

Level and prerequisites. L3 — integration level: the reader must be able to specify what preparation produces and to recognise the failure modes that come from the file formats rather than from the model. Prerequisites: the first sheet of this layer (whether retrieval belongs in the system at all).

Where to go next

References

  • Microsoft — Chunk documents in Azure AI Search — the token limit of an embedding model, the truncation warning, the fixed-size and semantic chunking strategies, and the overlap recommendation.
  • Pinecone — Chunking Strategies for LLM Applications — the definition of chunking, the two reasons it is necessary, and what exceeding an embedding model’s context window does to the data.
  • Anthropic — Introducing Contextual Retrieval — the standard pipeline’s chunk size in the pipeline’s own words.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.