Retrieval is the hard part of RAG

When a RAG system gives a bad answer, the first instinct is to blame the model or rewrite the prompt. In my experience the problem is usually upstream: the model never saw the right context.

The model can only use what you give it

A language model answering over your documents is doing reading comprehension. If the passage that contains the answer isn't in the context window, no amount of prompt engineering will help. So the first question for any bad answer is simple:

Was the right chunk retrieved?

If the answer is no, you have a search problem, not a generation problem.

Things that tend to help

Chunk on structure, not character counts. Splitting on headings and paragraphs keeps related sentences together.
Use hybrid search. Keyword search (BM25) catches exact identifiers and names that embeddings blur.
Log what you retrieved. You can't debug retrieval you can't see.

A small example of the last point:

def answer(question: str) -> str:
    chunks = retriever.search(question, k=8)
    log.info("retrieved", ids=[c.id for c in chunks], q=question)
    return llm.generate(question, context=chunks)

None of this is novel. It's plain search engineering, and it's where most of the quality comes from.