EPISODE 6 · RAG EVALUATION

THE EVAL SUITE

You built a RAG chatbot for an airline. It answers questions about baggage, refunds and pets. How do you know it works? These are the notes for the whole answer — every metric, where it is measured, and what it is actually asking.

recallprecisionfaithfulnessanswer relevancecorrectnesssafetyspeedcost

Watch the episode

THE PROBLEM

Why evals exist

Without evals, every change you make to a RAG system ends the same way: “Yeah… it feels better.” That is not a claim anyone can check, including you.

Evals exist so you can change your system without guessing. Change the model. Change the prompt. Change the chunk size. Then get an answer to three questions that have numbers attached:

WHAT GOT BETTER

↑

and by how much

WHAT GOT WORSE

↓

the trade you made

WHAT YOU BROKE

the one nobody noticed

THE WHOLE IDEA

A RAG system has parts. When the answer is wrong, you need to know which part was wrong — the search, or the writing. That is what the levels are for.

THE FOUNDATION

The golden set

Before any metric means anything, you need a fixed set of questions with known correct answers. In the episode it is fifty questions about SkyHigh’s policies, each one paired with the answer a human agrees is right, and each one tagged with the document chunks that should come back for it.

Two rules make the set worth having:

You never edit it to make a score look better. The moment you do, it stops being a ruler and starts being a mirror.
You run the same set every time. Comparing v6 to v7 only means something if both ran the identical questions.

WHERE IT GROWS

The set is not frozen forever — it grows from the bottom. Every real failure in production becomes a new row in it. That is the loop, and it is the last section on this page.

LEVEL 1 · COMPONENTS

Test the pieces, one at a time

A RAG system has two moving parts: a retriever that searches your documents, and a generator that writes an answer from what it found. Test them separately first, because a failure at this level tells you exactly who to blame.

The retriever — did it find the right pages?

You already know which chunks should come back for each golden question, so this is arithmetic. No LLM, no judgement, no cost.

CHUNK 1

found

relevant

CHUNK 2

found

relevant

CHUNK 3

junk

not relevant

CHUNK 4

junk

not relevant

CHUNK 5

missed

relevant

RECALL — 2 of 3

Of the chunks that should have come back, how many did?

PRECISION — 2 of 4

Of the chunks that did come back, how many were worth it?

Three chunks were relevant; four came back. Two of the three were found (recall), and two of the four were worth retrieving (precision).

Recallprogrammatic

Of the chunks that should have come back, how many actually did? Low recall means the answer was doomed before the model saw anything — the information simply was not in the room.

Precisionprogrammatic

Of the chunks that did come back, how many were relevant? Low precision means you handed the model a pile of noise and hoped it would pick correctly.

THE TRADE

These two pull against each other. Retrieve more chunks and recall goes up while precision goes down. That is not a bug to fix — it is the dial you are tuning, and regression testing is how you see which way you moved it.

The generator — did it write a good answer from them?

Now hand the model a known-good set of chunks and judge only the writing. If the retrieval was perfect and the answer is still wrong, the generator is the problem.

FaithfulnessLLM judge

Is every claim in the answer actually supported by the retrieved chunks? An unfaithful answer is one the model made up — fluent, confident, and not in your documents.

Answer relevanceLLM judge

Did the answer address the question that was asked? An answer can be perfectly faithful to the chunks and still not answer what the passenger wanted to know.

THESE TWO ARE NOT THE SAME

An answer can be faithful but irrelevant (everything it says is in the documents, none of it is what you asked) and relevant but unfaithful (it answers the question, using facts it invented). You need both numbers.

THE TOOL

Using an LLM as a judge

Recall and precision are counting. Faithfulness and relevance are not — there is no string to compare. So you give a second model the question, the chunks, the answer and a rubric, and ask it to score.

What makes a judge trustworthy

A strict rubric, not a vibe. “Rate 1–5” gives you noise. Spell out what each score means and what disqualifies an answer.
Show it the evidence. The judge sees the retrieved chunks, so “is this claim supported?” is a question it can actually answer.
Spot-check it against humans. The judge is a measuring instrument. Instruments get calibrated.

THE COST

An LLM judge is slower and more expensive than counting, which is exactly why it is reserved for the things counting cannot do — and why, in production, you only run it on a sample.

LEVEL 2 · PIPELINE

Test them working together

Both components can pass on their own and the system can still fail, because in Level 1 you fed the generator perfect chunks. Now let the real retriever feed the real generator and measure the thing the passenger actually gets.

Same fifty questions. Three relationships between three objects — the RAG triad:

Context relevance compares the question to what came back. Faithfulness compares the chunks to the answer. Answer relevance compares the question to the answer.

Context relevancequestion → chunks

Was what came back actually about the question? This is the only one of the three that needs no answer key at all — just the question and the chunks.

Faithfulnesschunks → answer

Is the answer grounded in what was retrieved, or did the model fill the gaps itself?

Answer relevancequestion → answer

Does the finished answer address what was asked?

COMPONENT vs PIPELINE

Component scores tell you a part’s ceiling — the best it can do under ideal conditions, and a good diagnostic when something breaks. Pipeline scores tell you reality. The pipeline number is the one you ship on.

LEVEL 3 · APPLICATION

Judge the finished product

The passenger does not care about recall. They care about one thing: “Did the chatbot give me a useful answer?” Level 3 stops measuring machinery and starts measuring the product.

Correctnessvs the golden answer

Is the answer actually right? This is the one that needs your golden set — you cannot score correctness without knowing the correct answer.

Completenessvs the golden answer

Did it cover everything it should have? A correct answer that stops halfway still sends the passenger to a phone line.

Styletone rubric

Does it sound like your airline? Correct and rude is still a support failure.

Safetyadversarial set

Does it leak PII, promise refunds it cannot authorise, or fall for a jailbreak? Safety is tested with its own set of questions designed to break it.

Operationsmeasured, not judged

Latency and cost per question. These are not quality metrics — they are the reason a technically excellent system still gets switched off.

CORRECT ≠ COMPLETE

“Yes, pets are allowed in the cabin” is correct. It is also missing the weight limit, the carrier requirement and the fee — so the passenger still ends up calling. Two separate numbers, because they fail separately.

PUTTING IT TOGETHER

So now you have an eval suite

The order is not arbitrary — each level explains failures in the next one.

Components

the pieces, one at a time

→

Pipeline

the pieces working together

→

Application

the finished product

Test the pieces. Then test them together. Then judge the final thing.

Run the whole thing and you get one report. When a number moves, the level it moved at tells you where to look.

AFTER EVERY CHANGE

Regression testing

You are going to change things — chunk size, the prompt, the model — to push scores up. The problem is that a change that helps one number usually hurts another, and you will not notice unless you re-run everything.

So: change one thing, re-run the whole suite on the same set, and log the result against a version number.

Worked example — v6 → v7

You cut the chunk size from 500 tokens to 200 to make retrieval sharper. Here is what the suite says happened:

METRICv6v7VERDICT

Recall0.820.89better

Precision0.780.71worse

Latency1.2s1.8sworse

One change. One improvement, one regression, one cost. Without the suite you would have shipped this knowing only the first line.

WHY THE LOG MATTERS

Three weeks later somebody asks why the bot got slower. With a version log you answer in ten seconds. Without one, you are reading commit messages and guessing.

This is the whole workflow: change → re-run → compare → decide. The decision is still yours. The suite just makes sure you are making it with evidence rather than a feeling.

IN PRODUCTION

Online evaluation

Everything above happens before the chatbot meets a passenger. Once it is live, the rules change: real people ask questions you never thought of, so there is no answer key for most of what comes in.

Someone asks whether they can bring a peacock. Someone asks whether the airline pays for a hotel after a storm cancellation. Neither is in your fifty.

What still works without an answer key

Two of the metrics never needed one. Both compare things you already have in hand:

FAITHFULNESS

✓

chunks vs answer

ANSWER RELEVANCE

✓

question vs answer

CORRECTNESS

—

needs the right answer

Correctness is the one you lose. You cannot automatically know whether an answer is right without knowing what right was.

So you watch behaviour instead

What you lose in ground truth you make up in volume. Real traffic tells you where the chatbot is failing, if you count the right things:

Thumbs up or down

the passenger told you directly

Asked the same thing again

the first answer did not land

Gave up and called

the expensive kind of failure

None of these prove an answer was wrong. All of them point at conversations worth reading.

Sample it

Running an LLM judge on every conversation is not sustainable. Judge a small percentage properly instead — the sample gets you the same signal at a fraction of the bill.

● logged & counted — free● judged by an LLM — 5% of the bill

Every conversation is logged and counted. A random slice gets the full LLM-judge treatment.

SELF-IMPROVING

The loop

Here is the part that makes all of it compound: your live application generates new test cases for you.

A passenger asks whether the airline will pay for their hotel. The chatbot says no. The passenger thumbs it down. That is not just a bad conversation — it is a new test case. You find the correct answer, add the pair to your golden set, and the next time you change anything, that failure is one of the things you test against.

Six stations. The seventh in the sequence is the first one again — which is the entire point.

THE COMPOUNDING BIT

Your real users show you what you missed. Every important failure becomes a test that protects you from making the same mistake again. A six-month-old eval suite is worth far more than a new one, and this is why.

ONE PAGE

Cheat sheet

Eight numbers, three levels, and what each one is actually asking.

RecallComponentsDid the right chunks come back?counted

PrecisionComponentsWas what came back worth it?counted

FaithfulnessPipelineIs the answer grounded in the chunks?LLM judge

Answer relevancePipelineDoes it answer the question?LLM judge

CorrectnessApplicationIs it right, vs the golden answer?LLM judge

SafetyApplicationDoes it leak or get jailbroken?adversarial set

SpeedApplicationHow long does a passenger wait?measured

CostApplicationWhat does one question cost?measured

And underneath, three questions

If you can only remember one thing from the episode, remember that all eight metrics collapse into these:

THE RETRIEVER

Did we find the right information?

THE GENERATOR

Did the model use it correctly?

IN PRODUCTION

Can we trust it in the real world?

THE POINT

Without evals, every change is “yeah, it feels better.” With evals, it is “here’s what actually changed.”

Notes for Vellumy — The Eval Suite.

The SkyHigh Airlines chatbot, its policy documents and every figure on this page are worked examples built for the episode.