Split the pipeline before you score it
A RAG answer depends on two independent steps. Retrieval decides which text the model sees. Generation decides what the model writes from that text. When an answer is bad, the cause is one of four things: the right chunk was never indexed, the right chunk was indexed but ranked below the cutoff, the chunk was in the context but the model ignored it, or the model added claims that the context does not support. Each cause has a different fix. An end-to-end score cannot tell them apart, so measure the steps separately.
Retrieval metrics
- Recall at k — the share of questions whose correct chunk appears in the top k results. This is the first number to fix, because generation cannot recover from a missing chunk.
- Precision at k — the share of retrieved chunks that are useful. Low precision wastes context window and adds distractors.
- Mean reciprocal rank — how high the first correct chunk sits. It rewards putting the best chunk first, which matters when the window is small.
- nDCG — a graded ranking score when several chunks are partly relevant.
Compute these against a labelled set: for each question you record the chunk identifiers that genuinely answer it. Without that label you are measuring the model's opinion of itself.
Generation metrics
- Faithfulness (groundedness) — the share of claims in the answer that the retrieved context supports. Break the answer into claims and check each one against the context, not against the question.
- Answer relevance — whether the answer addresses the question that was asked, independent of whether it is true.
- Context utilization — whether the answer used the retrieved evidence or ignored it and answered from parametric memory.
- Abstention — whether the system says it does not know when the context does not contain the answer. Score this explicitly, because a confident wrong answer is worse than a refusal.
Build the golden set from real questions
Collect 50 to 100 questions that users actually ask, with the wording they use. Add a few questions the corpus cannot answer, so abstention is measured. For each question store the gold chunk ids and a reference answer. Keep the set fixed between runs; when you add questions, report the score on the old subset as well, or you cannot compare runs.
Isolate the two stages
Two cheap experiments separate retrieval failures from generation failures. First, feed the generator the gold chunks and score the answers. If quality is high, retrieval is your bottleneck. Second, feed the gold answer's chunks through retrieval only and score recall. If recall is high but answers are wrong, the generator or the prompt is the bottleneck. Run both before you tune anything.
Set thresholds from your own baseline
Published targets such as "85% recall" are other people's corpora, chunk sizes and questions. Record the current score, change one thing, and keep the change only if the score moves in the right direction on the same set. A five point gain on a fixed set of 80 questions is meaningful; a five point gain after you rewrote the questions is not.
Track failures as a taxonomy, not a number
For every failing question, record one label: missing from index, ranked too low, ignored in context, or unsupported claim. The counts tell you where to spend effort. A pipeline with 90% recall and many unsupported claims needs prompt work, not a new vector database. A pipeline with 60% recall needs better chunking or a hybrid keyword search, not a better generator.
Online signals
Offline scores do not capture whether users trust the answers. Log which answers were followed by a rephrase, a second search or a click on a cited source. A rephrase within a few seconds is a strong signal that the answer missed. These signals are cheap to collect and they tell you which offline failures matter.