Embeddings for Engineers

The Problem#

Embeddings are often explained as magic arrays. Engineers need a practical mental model to decide what to embed, when to re-embed, and how to debug poor search results.

Why It Matters#

Embeddings are now used in docs search, support assistants, recommendation systems, duplicate detection, semantic routing, and RAG. The quality of those features depends less on hype and more on boring engineering: chunking, metadata, indexing, evaluation, and access control.

OpenAI, Google, Cohere, and many vector database providers expose embedding models, but the model is only one part of the system.

Project Example#

For a docs assistant, embed the chunk users should retrieve: a runbook section, API method, FAQ answer, or support resolution. Do not embed entire manuals as one vector, and do not embed tiny fragments that lose the surrounding instruction.

Implementation Example#

{
  "chunkId": "billing-runbook:retry-payments",
  "text": "Retry failed payment captures only after checking provider status...",
  "embeddingModel": "text-embedding-3-large",
  "dimension": 3072,
  "metadata": {
    "service": "billing",
    "visibility": "engineering",
    "version": "2026-08"
  }
}

Store the model and dimension because embeddings from different models are not comparable.

Implementation Checklist#

• Choose one embedding model per index.
• Store model name and dimension with every vector.
• Re-embed when the model or chunking strategy changes.
• Evaluate retrieval against known questions.
• Keep metadata for tenant, product, version, and permissions.
• Preserve source document links for citations.
• Combine vector search with keyword search for exact identifiers.
• Track retrieval failures reported by users.

Production Notes#

When retrieval is poor, inspect the returned chunks before changing the prompt. Most RAG failures start before generation: wrong chunk, stale document, missing metadata filter, or overly broad search.

Common Mistakes#

• Comparing vectors created by different models.
• Changing chunking without rebuilding the index.
• Assuming high similarity means the answer is correct.
• Forgetting exact keyword search for ids and error codes.
• Embedding data the current user is not allowed to retrieve.

How To Validate#

Create a small evaluation set from real questions. For each question, record the document or chunk that should be retrieved. When you change the embedding model, chunk size, or index settings, measure whether those expected chunks still appear in the top results.

Summary#

Embeddings encode useful similarity only when the chunking, metadata, model, and evaluation match the real product workflow.