Back to the library
software engineering
LLM Application Evaluation
Change a prompt and nobody can say whether it got better. This builds the eval set, the scorers and the CI check that answer that before you ship.
Starts with the eval set, because every metric downstream is only as honest as the cases it runs on.
Builds rubric scorers and an LLM judge, then holds them to the standard a judge has to meet: agreement with a human on cases you already labelled.
Finishes with regression testing and per-case diffs wired into CI, so a prompt change that quietly breaks nine cases stops being something you find out from a user.
Who it's for
- Engineers shipping an LLM feature with no way to compare two versions of it
- Teams whose prompt changes are argued about rather than measured
- Anyone who has been asked to prove an AI feature improved
Chapter outline
Foundations and the First Eval SetConstruct your first 20-case evaluation dataset in JSONL, define explicit criteria, and run a basic pass/fail harness against a live LLM endpoint to catch your first regression.6 Lessons · ~4.4h
- Constructing the 20-Case Evaluation Dataset in JSONL1h 17m
- Implementing the Live LLM Endpoint Pass/Fail Scorer1h 19m
- Catching Your First Prompt Regression with the Harness42m
- Concept canvas10m
- Assembling the Complete Miniature Evaluation Harness53m
- Practice checkpoint5m
Rubrics and LLM-as-a-Judge ScorersReplace vague pass/fail checks with multi-dimensional rubrics and implement an LLM-as-a-judge scorer that automates evaluation at scale.6 Lessons · ~3.7h
- Deconstructing Vague Quality Into Explicit Rubric Dimensions24m
- Implementing the LLM-as-a-Judge Scorer in Python1h 16m
- Calibrating Judge Agreement Against Human Verdicts56m
- Concept canvas10m
- Assembling the Complete Multi-Dimensional Scorer Harness50m
- Practice checkpoint5m
Dataset Expansion and Failure CategoriesExpand your eval set across production traces, synthetic adversarial variants, and off-topic inputs to catch regressions before they reach users.6 Lessons · ~2.7h
- Sampling and Cleaning Production Traces28m
- Generating Synthetic Adversarial Variants26m
- Practice checkpoint5m
- Tagging Failure Category Taxonomies53m
- Assembling the Diversified 80-Case Evaluation Suite45m
- Practice checkpoint5m
Regression Testing and Per-Case DiffsAfter this part you will be able to analyze before/after prompt changes using per-case diff reporting to identify regressions and improvements with category-level precision.5 Lessons · ~4.1h
- Implementing Per-Case Diff Reporting1h 29m
- Applying Category-Level Thresholds1h 1m
- Running Before and After Prompt Comparisons33m
- Practice checkpoint5m
- Isolating Regressions for Safe Deployments56m
CI Integration and Production Feedback LoopsAfter this part you will be able to wire your evaluation harness into CI/CD pipelines and set up production sampling loops to continuously close gaps in your eval dataset.5 Lessons · ~3.7h
- Automating Regression Gates with GitHub Actions26m
- Sampling Production Traffic and Triaging Low Scores25m
- Consolidating CI Thresholds and Feedback Loops1h 30m
- Practice checkpoint5m
- Closing Dataset Gaps with Live Production Failures1h 17m
Analyze: Evaluation dataset for an LLM application, select appropriate metrics…Build an evaluation dataset for an LLM application, select appropriate metrics, and analyze the results to identify potential regressions.4 Lessons · ~2.7h
- Phase 1: Foundation — LLM Eval Concepts and Conventions44m
- Phase 2: Scaffold — Evaluation Brief Outline and Dataset Plan30m
- Phase 3: Part — Dataset Construction44m
- Phase 4: Polish + Finalize — Complete Data-Analysis Brief44m