Context engineering beyond basic RAG

Most first RAG demos do the same three things. Split documents on a fixed width, embed the pieces, and paste the nearest neighbours into a prompt. That demo is useful. It is not context engineering. Context engineering is the discipline of deciding which text the model sees, in what order, with what instructions, and how you know the selection was wrong.

The model only conditions on what you hand it

A language model continues from the tokens in the prompt. It does not reach into your database on its own. If the right passage never enters the prompt, the model is guessing from pretraining. If the prompt is stuffed with near misses, the model has to pick a needle out of a pile of plausible text. Both failure modes look like 'the model is bad'. Often the retrieval step is the thing that failed.

Chunking is a product decision

A chunk is the unit you can retrieve. If you cut through the middle of a procedure, the retrieved span may contain the step without the condition that makes the step valid. If you keep whole chapters, the embedding averages many topics and the neighbour search gets vague. We treat chunk boundaries as something you choose against the shape of the corpus: headings, tickets, clauses, conversation turns. Overlap is a way to keep a sentence that sits on a boundary, not a default you copy from a tutorial. Parent and child chunks are useful when a short span should be found, but the model should read the surrounding section. None of these choices is universal. The corpus decides.

Embeddings are one signal

Dense embeddings are good at paraphrase. They are weak when the user asks for an exact identifier, a product code, an error string, or a name that appears once. Sparse lexical search still wins those queries. Hybrid search runs both and merges the lists. Reciprocal rank fusion is a simple merge: each list votes, and you do not have to pretend the two scores live on the same scale. The point is not a clever formula. The point is that one retrieval family will systematically miss a class of questions your users actually ask.

Reranking is where precision is recovered

The first-stage search should be wide enough to include the right passage. It will also include distractors. A reranker reads the query and a short list of candidates and orders them again. That second read is more expensive than a vector lookup, so it belongs on a shortlist, not on the whole corpus. After reranking, you still decide how many passages to place in the prompt, and you decide whether to show them as quotes the model must stick to. We would rather give the model fewer passages that a person can audit than a long paste no one will read.

Retrieval has to be evaluated on its own

If you only score the final answer, you cannot tell a retrieval miss from a generation miss. A retrieval eval is a set of questions with the passages that should have been found. You can mark a hit when the right document, or the right span, appears in the shortlist before the model speaks. When it does not, you fix chunking, metadata filters, or the query itself. Query rewriting helps when users write the way they talk and the corpus is written the way a manual is written. Filters help when the corpus has tenants, products, or dates and the question implies one of them. You do not need a public benchmark to do this. You need questions from the real task and a person who knows which document is the answer.

What goes into the prompt, and what stays out

Context is not only retrieved text. It is the instruction, the tool results, the conversation so far, and any policy you refuse to violate. Order matters because the model attends to all of it. We keep instructions stable, put retrieved text in a delimited block, and tell the model to say when the block does not contain the answer. That last instruction is a product decision: an abstention is better than a fluent invention. Metadata such as titles and dates belongs next to the passage so the model can prefer the current document when two passages disagree.

def build_context(query, corpus):
    lexical = sparse_search(query, corpus)
    dense = vector_search(embed(query), corpus)
    shortlist = fuse(lexical, dense)
    ranked = rerank(query, shortlist)
    passages = ranked[:K_FOR_PROMPT]
    if not covers(query, passages):
        return abstain()
    return prompt(instructions, passages, query)

A practical order of work

• Collect questions the system must answer, and the documents that hold those answers.
• Chunk in a way that preserves the unit a person would quote.
• Run lexical and dense search, then fuse.
• Rerank a shortlist and cap what enters the prompt.
• Score retrieval on its own before you blame the generator.
• Teach the model to abstain when the context is empty of the answer.

Basic RAG is a pipeline. Context engineering is the set of decisions that make that pipeline honest: what you index, what you retrieve, what you refuse to paste, and how you notice a miss. The model is the last step, not the strategy.

Failure modes we look for in review

A retrieved passage can be topically related and still be the wrong policy version. Metadata filters and dates exist for that case. A passage can be right and still be too long for the instruction to survive, so we cap the prompt and keep the instruction outside the retrieved block. A question can be unanswerable from the corpus. The correct behaviour is abstention, with the shortlist available to the person who has to decide whether the corpus is incomplete. We also watch queries that return the same wrong neighbour over and over. That pattern means the embedding, the chunk boundary, or the query itself is biased, and another round of prompt wording will not fix it. Context engineering ends when a reviewer can point at the passage that should have been used and see it in the trace, or see a clear reason it was not retrieved.