Evaluation Pipelines

Course overview

Evaluation pipelines for RAG: synthetic dataset generation, retrieval metrics (recall, precision, MRR, nDCG) and generation metrics (faithfulness, helpfulness, correctness), plus LLM-as-judge calibration and bias. Practical approach to measure and compare systems.

Shows how to create synthetic datasets tailored to RAG test cases for reproducible diagnostics.

Defines retrieval metrics — recall, precision, MRR, nDCG — and how to interpret them for coverage and ranking evaluation.

Covers generation metrics focused on faithfulness and usefulness, and discusses using an LLM as judge including calibration challenges and bias considerations.