RAG Evaluation in CI: Fail the Build Before Your Users Find the Regression

1. The Problem: RAG Fails Without an Error

A RAG pipeline has two stages that fail in different ways. Retrieval can miss the passage that holds the answer, or bury it under noise. Generation can ignore the passage, embellish it or answer a different question. The user sees one symptom — a bad answer — and eyeballing five queries after a change cannot tell you which stage broke, or whether yesterday’s tweak made things better or worse.

A CI gate is that idea made boring: a fixed set of questions with reviewed answers runs through your real pipeline on every relevant change, metrics are computed per stage, results are compared with a baseline that carries a noise margin, and the job exits non-zero when quality drops. It will not make your system good. It makes sure you find out when it stops being as good as it was.

2. Minimum Concepts Before Touching Code

Six ideas are enough. The metric names below are the ones in the Ragas 0.4.3 collections API (ragas.metrics.collections). Note the naming trap: the docs page calls the last metric Response Relevancy, the 0.4 class is AnswerRelevancy, and the old class you will meet in older tutorials is ResponseRelevancy.

3. Step 1 — Build a Golden Set of ~50 Questions (Humans Included)

Fifty questions is a smoke detector, not a benchmark: each one is 2% of the mean, so the set is big enough to catch a broken pipeline and too small to rank two good ones. Compose it on purpose: real questions from your logs, questions written by someone who knows the domain, and a few synthetic drafts to widen coverage. The Ragas generator’s default mix is 50% single-hop specific and 25% each of multi-hop abstract and multi-hop specific; yours should add what generated sets tend to under-represent — messy real phrasing and questions the corpus cannot answer.

JSONL — two reviewed golden records (invented content, one object per line)

{"id": "q-014", "kind": "multi_hop", "origin": "user_log", "question": "Can I still get a refund on an item that arrived damaged after the 30-day window?", "reference": "Yes. Items damaged on arrival are refundable after 30 days if you report them within 14 days of delivery and attach a photo.", "expected_doc_ids": ["policies/refunds.md", "policies/shipping-damage.md"], "reviewed_by": "maria", "reviewed_on": "2026-10-12"}
{"id": "q-031", "kind": "unanswerable", "origin": "sme", "question": "Do you price-match competitors during Black Friday?", "reference": "The documentation does not cover price matching; the assistant should say it cannot answer.", "expected_doc_ids": [], "reviewed_by": "maria", "reviewed_on": "2026-10-12"}

Over-generate drafts with the Ragas TestsetGenerator, then review every row by hand. In the current docs the generator still takes a LangChain-wrapped LLM plus Ragas’s own OpenAIEmbeddings, and exposes generate_with_langchain_docs(documents, testset_size). I build the document list myself so there is no extra loader dependency to install.

Python — draft candidates with TestsetGenerator (ragas 0.4.3)

# evals/make_drafts.py  - draft candidates only; nothing here is "golden" yet
import os
from pathlib import Path

import openai
from langchain_core.documents import Document
from langchain_openai import ChatOpenAI
from ragas.embeddings import OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.testset import TestsetGenerator

docs = [
    Document(page_content=p.read_text(encoding="utf-8"), metadata={"source": str(p)})
    for p in Path("docs").rglob("*.md")
]

generator = TestsetGenerator(
    llm=LangchainLLMWrapper(ChatOpenAI(model=os.environ["GEN_MODEL"])),
    embedding_model=OpenAIEmbeddings(client=openai.OpenAI()),
)

# over-generate: ~80 drafts to end up with ~50 that survive human review
testset = generator.generate_with_langchain_docs(docs, testset_size=80)
testset.to_pandas().to_csv("evals/drafts.csv", index=False)
print("Now open evals/drafts.csv and review EVERY row by hand.")

Then make the set defend itself. A check with no LLM calls runs first in CI: required fields, unique ids, known kinds, and — the one that saves you — expected_doc_ids that still exist in the index. Golden rows rot when the corpus changes; this makes the rot loud instead of silently lowering recall.

Python — evals/check_golden.py: integrity before spending tokens

# evals/check_golden.py  - no LLM calls, runs in seconds, fails fast
import json
import sys
from pathlib import Path

from app.rag import known_doc_ids  # YOUR code: set of doc ids currently in the index

REQUIRED = {"id", "question", "reference", "kind", "reviewed_by", "reviewed_on"}
KINDS = {"single_hop", "multi_hop", "unanswerable", "messy_query"}

rows = [json.loads(l) for l in Path("evals/golden.jsonl").read_text(encoding="utf-8").splitlines() if l.strip()]
index, errors, seen = known_doc_ids(), [], set()

for r in rows:
    if missing := REQUIRED - r.keys():
        errors.append(f"{r.get('id')}: missing {sorted(missing)}")
    if r.get("id") in seen:
        errors.append(f"{r['id']}: duplicate id")
    seen.add(r.get("id"))
    if r.get("kind") not in KINDS:
        errors.append(f"{r.get('id')}: unknown kind {r.get('kind')!r}")
    for doc_id in r.get("expected_doc_ids", []):
        if doc_id not in index:
            errors.append(f"{r['id']}: stale doc id {doc_id} (corpus changed - re-review)")

print(f"{len(rows)} golden rows, {len(errors)} problems")
for e in errors:
    print("::error title=golden set::" + e)
sys.exit(1 if errors else 0)

4. Step 2 — Score Your Pipeline with Ragas 0.4.3

Pin the version: pip install ragas==0.4.3. On 2026-10-14 that is the newest release on PyPI (published 2026-01-13) and the newest tag on GitHub. The 0.4 line moved metrics to ragas.metrics.collections: you build a metric from an LLM created with llm_factory, call await metric.ascore(...) with plain keyword arguments, and read .value from the returned MetricResult. The v0.4 migration guide lists evaluate() as deprecated and the old ragas.metrics classes are marked for removal in 1.0, so older tutorials will not match. Owning a small loop also gives you what a CI gate needs: control over concurrency, NaNs and exit codes. The eval needs exactly one thing from your system:

Python — app/rag.py: the contract (names are placeholders for your code)

# app/rag.py - the only thing the eval needs from your system
def answer_question(question: str) -> dict:
    """Must run the SAME path production runs: retrieve -> rerank -> build prompt -> generate."""
    chunks = retrieve_and_rerank(question)          # your code
    prompt_chunks = fit_to_context_window(chunks)   # whatever really ends up in the prompt
    answer = generate(question, prompt_chunks)      # your code
    return {
        "answer": answer.text,
        "contexts": [c.text for c in prompt_chunks],   # EXACTLY what the model saw, in prompt order
        "doc_ids": [c.doc_id for c in prompt_chunks],
        "refused": answer.refused,                      # did it say "I cannot answer this"?
    }

The line that matters is contexts: pass the chunks that actually went into the prompt, after reranking and truncation, not the raw top-k you fetched. Scoring chunks the model never saw gives you precision numbers about a pipeline you do not run. Here is the harness, with argument names checked against the 0.4.3 docs and source:

Python — evals/run_ragas.py: one pipeline run, four metrics per question

# evals/run_ragas.py  - argument names checked against the ragas==0.4.3 docs and source
import asyncio
import json
import math
import os
from pathlib import Path

from openai import AsyncOpenAI
from ragas.cache import DiskCacheBackend
from ragas.embeddings.base import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    AnswerRelevancy,
    ContextPrecision,
    ContextRecall,
    Faithfulness,
)

from app.rag import answer_question  # YOUR pipeline -> {"answer", "contexts", "doc_ids", "refused"}

JUDGE_MODEL = os.environ["RAGAS_JUDGE_MODEL"]  # a pinned, dated snapshot - never a floating alias
EMBED_MODEL = os.environ.get("RAGAS_EMBED_MODEL", "text-embedding-3-small")
CONCURRENCY = int(os.environ.get("RAGAS_CONCURRENCY", "6"))

# metric -> how its ascore() is called (argument names verified in the 0.4.3 source)
CALLS = {
    "faithfulness": lambda m, q, a, c, ref: m.ascore(user_input=q, response=a, retrieved_contexts=c),
    "context_precision": lambda m, q, a, c, ref: m.ascore(user_input=q, reference=ref, retrieved_contexts=c),
    "context_recall": lambda m, q, a, c, ref: m.ascore(user_input=q, retrieved_contexts=c, reference=ref),
    "answer_relevancy": lambda m, q, a, c, ref: m.ascore(user_input=q, response=a),
}


def build_metrics():
    client = AsyncOpenAI(max_retries=4)  # built per run: one client per event loop
    cache = DiskCacheBackend(cache_dir=".ragas_cache") if os.environ.get("RAGAS_CACHE") == "1" else None
    llm = llm_factory(JUDGE_MODEL, client=client, cache=cache, temperature=0)
    emb = embedding_factory("openai", model=EMBED_MODEL, client=client, cache=cache)
    return {
        "faithfulness": Faithfulness(llm=llm),
        "context_precision": ContextPrecision(llm=llm),
        "context_recall": ContextRecall(llm=llm),
        "answer_relevancy": AnswerRelevancy(llm=llm, embeddings=emb),
    }


def clean(value):
    """NaN means 'the judge produced nothing' - keep it as None, never as 0."""
    v = float(value)
    return None if math.isnan(v) else v


def doc_recall(expected, got):
    """Deterministic, no LLM: which expected source docs reached the prompt?"""
    return len(set(expected) & set(got)) / len(set(expected)) if expected else None


async def score_row(row, metrics, sem):
    async with sem:
        out = await asyncio.to_thread(answer_question, row["question"])
        rec = {"id": row["id"], "kind": row["kind"], "answer": out["answer"],
               "n_ctx": len(out["contexts"]),
               "doc_recall_at_k": doc_recall(row.get("expected_doc_ids", []), out["doc_ids"])}
        if row["kind"] == "unanswerable":  # the right behaviour is a refusal: not a Ragas question
            rec["refusal_ok"] = 1.0 if out.get("refused") else 0.0
            return rec
        q, ref, ans, ctxs = row["question"], row["reference"], out["answer"], out["contexts"]
        if not ctxs:  # retrieval returned nothing: that IS the fault, and Ragas would raise
            return {**rec, "context_recall": 0.0, "context_precision": 0.0,
                    "faithfulness": None, "answer_relevancy": None}
        names = list(CALLS)
        results = await asyncio.gather(
            *(CALLS[n](metrics[n], q, ans, ctxs, ref) for n in names), return_exceptions=True
        )
        errors = []
        for name, res in zip(names, results):
            if isinstance(res, Exception):
                rec[name] = None
                errors.append(f"{name}: {type(res).__name__}")
            else:
                rec[name] = clean(res.value)
        rec["errors"] = errors
        return rec


async def run_once(golden_path="evals/golden.jsonl"):
    lines = Path(golden_path).read_text(encoding="utf-8").splitlines()
    rows = [json.loads(l) for l in lines if l.strip()]
    metrics, sem = build_metrics(), asyncio.Semaphore(CONCURRENCY)
    scored = await asyncio.gather(*(score_row(r, metrics, sem) for r in rows))
    return {"judge_model": JUDGE_MODEL, "n": len(scored), "rows": scored}

5. Step 3 — Separate Retrieval Faults from Generation Faults, Then Set Thresholds

A falling average is not a diagnosis. The classifier below assigns each question to one stage and blames retrieval first: if the needed evidence never reached the prompt, a low faithfulness score is a symptom, not the cause. Combined with a deterministic doc-recall check (no LLM: did the expected source documents appear among the retrieved doc ids?), it tells you whether to open the index config or the prompt.

Python — evals/faults.py: one fault class per question

# evals/faults.py  - which STAGE broke, per question
from collections import Counter, defaultdict

ROW_BAD = 0.5  # per-question "this one failed" cutoff: a placeholder, tune it on reviewed failures


def low(value, cutoff=ROW_BAD):
    return value is not None and value < cutoff


def classify(r):
    """Order matters: blame retrieval first. If the evidence never reached the prompt,
    a low faithfulness score is a symptom, not the cause."""
    if r["kind"] == "unanswerable":
        return "ok" if r.get("refusal_ok") == 1.0 else "should_have_refused"
    if r["n_ctx"] == 0 or r.get("doc_recall_at_k") == 0.0 or low(r.get("context_recall")):
        return "retrieval_miss"        # needed evidence not in the prompt -> index, query, top_k, embeddings
    if low(r.get("context_precision")):
        return "retrieval_noise"       # evidence there but buried -> reranker, chunk size, top_k
    if low(r.get("faithfulness")):
        return "generation_unfaithful"  # evidence there, answer drifted -> prompt, model, temperature
    if low(r.get("answer_relevancy")):
        return "generation_off_topic"   # grounded but not answering the question -> prompt/format
    return "ok"


def fault_report(rows):
    by_class, ids = Counter(), defaultdict(list)
    for r in rows:
        c = classify(r)
        by_class[c] += 1
        if c != "ok":
            ids[c].append(r["id"])
    return by_class, ids

Now thresholds. Two rules keep a 50-question gate honest. First, gate on movement from a baseline, not on an absolute number you invented: the baseline stores each metric’s mean and its run-to-run noise (the standard deviation across repeated full runs), and the fail line sits K noise-units below it, never closer than a minimum margin. Second, add a hard floor only after you have seen real numbers — it is a product decision, so the config below ships with none. Because borderline results are exactly where judge noise bites, a WARN or FAIL triggers two more runs and the gate decides on the median of three, never on the best one.

Python — evals/gate.py: pass / warn / fail with a noise band

# evals/gate.py  - exit 0 = pass/warn, 1 = quality regression, 2 = harness broken
import argparse
import asyncio
import json
import math
import os
import statistics
import sys
from pathlib import Path

from evals.faults import fault_report
from evals.run_ragas import run_once

RAGAS = ["context_recall", "context_precision", "faithfulness", "answer_relevancy"]
METRICS = RAGAS + ["refusal_rate"]
HARD_FLOOR = {m: None for m in METRICS}  # product decision: fill in AFTER your first baseline
MIN_MARGIN = 0.03   # smallest drop we ever call a regression (a starting point, not a law)
K_NOISE = 3         # fail only when we are K noise-units below baseline
MAX_MISSING = 0.10  # more than 10% judge failures = broken harness, not a quality problem


def mean_and_missing(rows, key):
    vals = [r[key] for r in rows if r.get(key) is not None]
    mean = statistics.fmean(vals) if vals else math.nan
    return mean, (1 - len(vals) / len(rows)) if rows else 1.0


def summarize(run):
    answerable = [r for r in run["rows"] if r["kind"] != "unanswerable"]
    refusals = [r for r in run["rows"] if r["kind"] == "unanswerable"]
    out = {m: mean_and_missing(answerable, m) for m in RAGAS}
    if refusals:
        out["refusal_rate"] = mean_and_missing(refusals, "refusal_ok")
    return out


def decide(metric, value, base):
    margin = max(K_NOISE * base["noise"], MIN_MARGIN)
    fail_below = max(HARD_FLOOR[metric] or 0.0, base["mean"] - margin)
    warn_below = base["mean"] - max(base["noise"], MIN_MARGIN / 2)
    if not value >= fail_below:  # written this way so NaN FAILS instead of slipping through
        return "FAIL", fail_below
    if value < warn_below:
        return "WARN", warn_below
    return "PASS", warn_below


def judge(runs, baseline):
    per_run = [summarize(r) for r in runs]
    verdicts = {}
    for m in per_run[0]:
        if any(s[m][1] > MAX_MISSING for s in per_run):
            return None  # too many judge failures: the harness is broken, say so
        value = statistics.median(s[m][0] for s in per_run)  # median of 1 or 3 runs
        verdicts[m] = (value, *decide(m, value, baseline["metrics"][m]))
    return verdicts


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--golden", default="evals/golden.jsonl")
    ap.add_argument("--baseline", default="evals/baseline.json")
    ap.add_argument("--out", default="results")
    args = ap.parse_args()
    baseline = json.loads(Path(args.baseline).read_text(encoding="utf-8"))
    if baseline["judge_model"] != os.environ["RAGAS_JUDGE_MODEL"]:
        print("::error title=judge changed::judge model differs from the baseline judge; re-baseline first")
        sys.exit(2)

    runs = [asyncio.run(run_once(args.golden))]
    verdicts = judge(runs, baseline)
    if verdicts and any(v[1] != "PASS" for v in verdicts.values()):
        runs += [asyncio.run(run_once(args.golden)) for _ in range(2)]  # escalate: gate on median of 3
        verdicts = judge(runs, baseline)

    Path(args.out).mkdir(exist_ok=True)
    Path(args.out, "runs.json").write_text(json.dumps(runs, indent=2), encoding="utf-8")
    if verdicts is None:
        print("::error title=RAG eval harness::too many judge failures (see results/runs.json)")
        sys.exit(2)

    by_class, ids = fault_report(runs[-1]["rows"])
    lines = ["| metric | value | verdict | threshold |", "|---|---|---|---|"]
    lines += [f"| {m} | {v:.3f} | {d} | {t:.3f} |" for m, (v, d, t) in verdicts.items()]
    lines += ["", f"fault classes (last run): {dict(by_class)}"]
    lines += [f"- {c}: {', '.join(i)}" for c, i in ids.items()]
    summary = "\n".join(lines)
    print(summary)
    if os.environ.get("GITHUB_STEP_SUMMARY"):
        with open(os.environ["GITHUB_STEP_SUMMARY"], "a", encoding="utf-8") as fh:
            fh.write(summary + "\n")
    failed = [m for m, (_, d, _) in verdicts.items() if d == "FAIL"]
    if failed:
        print(f"::error title=RAG eval gate::regression in {', '.join(failed)}")
        sys.exit(1)


if __name__ == "__main__":
    main()

6. Step 4 — Fail the Build in GitHub Actions

The workflow stays small because the logic lives in Python. It runs when retrieval code, prompts or the eval itself change, and nightly on the default branch. The Python gate writes a Markdown table to the job summary (GITHUB_STEP_SUMMARY), emits ::error annotations, and the raw per-question results are uploaded as an artifact so a failing PR can be debugged without paying for a re-run.

YAML — .github/workflows/rag-eval.yml

name: rag-eval-gate

on:
  pull_request:
    paths:                       # only when something that can move quality changes
      - "app/rag/**"
      - "prompts/**"
      - "evals/**"
      - "requirements*.txt"
  workflow_dispatch:             # maintainers can run it on a fork PR by hand
  schedule:
    - cron: "17 3 * * *"         # nightly on the default branch

permissions:
  contents: read

concurrency:
  group: rag-eval-${{ github.ref }}
  cancel-in-progress: true

jobs:
  rag-eval:
    runs-on: ubuntu-latest
    timeout-minutes: 25
    # fork PRs get no secrets: skip them instead of turning them red
    if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
    env:
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      RAGAS_JUDGE_MODEL: ${{ vars.RAGAS_JUDGE_MODEL }}   # a dated snapshot, checked against the baseline
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt "ragas==0.4.3"
      - name: Golden set integrity (no LLM calls)
        run: python -m evals.check_golden
      - name: RAG eval gate
        run: python -m evals.gate
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: rag-eval-results
          path: results/

Record the baseline on purpose, never inside the PR job: five full runs with the cache off, from a clean main, committed through a reviewed PR. The standard deviation across those runs is the noise your gate must tolerate — it includes judge variance and generator variance, because the whole pipeline is re-run every time. The judge model comes from a repository variable and is compared with the one stored in the baseline, so it cannot change unnoticed.

Python — evals/baseline.py: measure the noise once, then commit it

# evals/baseline.py  - run on purpose, cache OFF; commit the result through a reviewed PR
import asyncio
import json
import os
import statistics
from pathlib import Path

from evals.gate import summarize
from evals.run_ragas import JUDGE_MODEL, run_once

REPEATS = 5
assert os.environ.get("RAGAS_CACHE") != "1", "the cache hides judge noise: turn it off for baselines"

runs = [asyncio.run(run_once("evals/golden.jsonl")) for _ in range(REPEATS)]
per_run = [summarize(r) for r in runs]
metrics = {}
for m in per_run[0]:
    means = [s[m][0] for s in per_run]  # one mean per full pipeline run
    metrics[m] = {"mean": statistics.fmean(means), "noise": statistics.stdev(means)}

Path("evals/baseline.json").write_text(
    json.dumps({"judge_model": JUDGE_MODEL, "repeats": REPEATS, "metrics": metrics}, indent=2),
    encoding="utf-8",
)
print("baseline written - review it, then open a PR")

7. A Cadence That Keeps the Gate Trustworthy

A gate decays unless someone feeds it. This is the routine that keeps mine from turning into a muted check:

8. Common Errors (All Preventable)

Almost every broken RAG gate I have debugged falls into one of these buckets. The left column is what gates people keep trusting; the right column is what gets them muted.

Conclusion

You now have the whole loop: a human-reviewed golden set, Ragas 0.4.3 collections metrics scored per question, a classifier that blames retrieval before generation, a baseline-relative gate with a noise band, and a GitHub Actions job that fails the build. None of the numbers in this post are results — they are slots for yours.

Start small this week: take 20 real questions from your logs, write the references by hand, run the harness locally with the cache on and read the fault classes before you look at any average. The first useful output is rarely a score; it is the list of questions your pipeline obviously fails — and that list is your backlog.

Sources

• Ragas docs — Available metrics (collections API names)
• Ragas docs — Faithfulness (formula, collections API)
• Ragas docs — Context Precision
• Ragas docs — Context Recall
• Ragas docs — Response / Answer Relevancy (needs embeddings)
• Ragas docs — Migrating from v0.3 to v0.4 (collections API, evaluate() deprecated)
• Ragas docs — Generate a RAG test set (TestsetGenerator)
• Ragas docs — Align an LLM judge with human labels
• PyPI — ragas (release history, 0.4.3)
• Ragas on GitHub — source and tags (v0.4.3)
• Zheng et al. — Judging LLM-as-a-Judge with MT-Bench (position, verbosity, self-enhancement bias)
• GitHub Docs — Events that trigger workflows (fork PRs, secrets, pull_request_target)
• GitHub Docs — Workflow commands (job summary, error annotations)
• DeepEval — RAG evaluation in CI (assert_test, deepeval test run)