Articles

Embeddings and semantic similarity: distance as a proxy for meaning

An embedding turns meaning into geometry; that geometry is a proxy with known limits.

Reading: 6 minAutomation & AI

Article cover: Embeddings and semantic similarity: distance as a proxy for meaning

An embedding model turns text into a vector, arranged so that texts a human would call similar end up near each other. The geometric metaphor is precise enough to build on and loose enough to mislead: what an embedding encodes is that model’s notion of similarity, learned from its training, not meaning itself.

Sentence-BERT is the work that made this practical — it turned a comparison that took “~65 hours” with BERT into one that takes “~5 seconds with SBERT” for the same collection of sentence pairs, by producing embeddings that can be computed once and then compared with a similarity measure. Everything in this layer depends on that shift: retrieval is only possible because similarity became a lookup instead of a computation over every pair.

Distance, and what it means

An embedding store exposes several distances because “similar” has more than one arithmetic reading. The vector extension used throughout these sheets supports “L2 distance, inner product, cosine distance, L1 distance, Hamming distance, and Jaccard distance”, each with its own operator — and the difference is not cosmetic: the same function is documented with a caveat for one of them (“<#> returns the negative inner product since Postgres only supports ASC order index scans on operators”), which is the kind of detail that silently changes a ranking when it is ignored.

Two consequences worth stating outright:

  • Distance is relative to the model. Two embeddings from different models are not comparable, and changing the embedding model means re-embedding the corpus.
  • Similarity is not relevance. The nearest neighbours of a query are the texts the model places nearest, which is a useful approximation of relevance and not a definition of it. That gap is what the reranking sheet and the evaluation sheets exist to measure.

Dense and sparse are complements

The dense passage retrieval work made the distinction explicit: retrieval “is usually implemented with an inverted index and can be seen as representing the question and context in high-dimensional, sparse vectors”, with sparse models “such as TF-IDF or BM25” as the de facto method; dense representations were shown to work alone, and the paper’s framing is that the two are complements — “the dense, latent semantic encoding is complementary to sparse representations by design. For example, synonyms or paraphrases”.

That sentence is the reason hybrid search exists as a technique rather than as a hedge: dense retrieval solves the vocabulary mismatch that breaks keyword search, and sparse retrieval solves the exact-token matching that dense retrieval blurs. The filters-and-hybrid-search sheet puts them together.

Decisions that belong to the embedding model

  • Dimensionality. A vector’s width is a storage and index decision: the relational extension stores half-precision vectors “up to 4,000 dimensions”, and the index types have their own limits — the index sheet takes this further.
  • Symmetric versus asymmetric. Some embeddings are trained so that a short query and a long document can be compared directly; others expect a query-style input against a passage-style input. Retrieval quality depends on using the model as it was trained, which is why the model’s own documentation — not the vector store’s — is the authority on how to embed queries and documents.
  • Normalisation. Whether vectors are unit-length decides whether cosine distance and inner product agree. Getting this wrong is invisible until ranking looks strange.
  • What is embedded. A chunk, a chunk plus its context, or a document summary: the same model with different inputs produces different retrieval behaviour, which is exactly the contextual-retrieval technique described in the chunking sheet.

What an embedding leaks

An embedding is often treated as a de-identified representation of text. It is not. An inversion attack reconstructs the text from its vector: the work titled Text Embeddings Reveal (Almost) As Much As Text shows an iterative method that generates candidate text and re-embeds it until the vectors match, and states that it is “able to recover 92%” of the input for 32-token text inputs, exactly.

The operational consequences are concrete and belong to two other sheets in this layer: an embedding store holds personal and confidential data and needs the access control of an index (the access-control sheet), and a corpus of vectors leaving the boundary is a disclosure of the corpus, not a summary of it.

Where this sits in the pipeline

Embedding is one step: text in, vectors out, once at indexing time and once per query. Smoothing over that step is how pipelines acquire their most persistent defects — a query embedded with the wrong input style, a corpus half-embedded with the previous model, vectors that were never normalised. The store cannot detect any of these; only a measurement can (see the evaluation sheets), which is why the L2 rule about reporting a performance figure with its conditions applies to retrieval quality as much as to tokens per second.

Level and prerequisites. L3 — integration level: the reader must be able to choose an embedding model and its input conventions, and to know which of the remaining problems belong to the index, the filters and the evaluation. Prerequisites: the L1 sheets on tokens and on how a model generates text, and the previous sheets of this layer (chunking decides what gets embedded).

Where to go next

References

  • Reimers and Gurevych — Sentence-BERT — the cost comparison that made embedding-based retrieval practical.
  • Karpukhin et al. — Dense Passage Retrieval — the sparse/dense complementarity and the de facto sparse baseline.
  • Morris et al. — Text Embeddings Reveal (Almost) As Much As Text — embedding inversion.
  • pgvector — README — the distances an embedding store exposes, the dimension limits and the operator caveat.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.