How does the LLM Evals Playbook work? I explain it clearly with a real evaluation pipeline

A real review process

Last week, I was organizing a regression test for a customer service Q&A agent. The most troublesome question was: after each model fine-tuning, how can you quickly tell which areas have improved and which have actually worsened?

Traditional unit testing is not suitable—LLM outputs are natural language and do not have fixed assertive values. After looking through various tools, I finally chose the idea of LLM Evals Playbook. It is not a single PDF, but an actionable evaluation framework. The core process is: defining evaluation criteria → generating test datasets → running evaluations → analyzing results and iterating.

Below is the first complete link I ran, with code and configuration files corresponding to each step.

Step One: Define the evaluation criteria

Evaluations must start from the business perspective. My scenario is customer service Q&A, so I defined three dimensions:

• Answer Accuracy: Whether the answer is based on the provided knowledge base and not the answer is uncertain.
• Integrity: Whether all subissues of the user's problem are covered.
• Tone compliant: Whether you remain friendly and don't shirk responsibility.

Each dimension is scored on a scale from 1 to 5, with an explanation of the evaluation criteria. Playbooks require these standards to be actionable; for example, "accuracy" should not be just "correct" but written as "every piece of information in the answer must be traceable in the original text in the knowledge base."

The most common mistake here is standards that are too vague. My first version was written as "Answers Should Be Accurate," but the scores given by different evaluators varied greatly. Later, it was changed to "If the answer contains entities or data not listed in the knowledge base, deduct 2 points," which improved consistency.

Step 2: Generate the test dataset

Playbook recommends that the dataset include three types of samples: normal problems, edge problems (e.g., missing context), and adversarial problems (e.g., induced models to produce inappropriate content).

I generated 50 normal questions using real user logs from my own business, then manually constructed 20 edge questions and 10 adversarial questions. Be careful when constructing adversarial questions—don't copy prompt injection templates directly from the internet, as they often stray from your actual business scenarios.

A Key Failure Point: The dataset must be aligned with the evaluation criteria. If your standard requires "not hallucinate," but most questions in the dataset are simple queries and the model won't make up baseless claims, then the evaluation results lack distinctiveness. The correct approach is to run a small sample run first, see where the model is prone to errors, and then supplement the dataset accordingly.

Step Three: Operational Evaluation

I connected to the Playbook using LangChain's Evaluator framework. The evaluation process for each test sample is: first, the Agent being evaluated answers questions, then an Evaluator LLM (such as GPT-4o) scores them according to preset criteria.

Scoring must be accompanied by reasons; it cannot be just a score. For example: "Accuracy 4 points: The answer mentioned the price was 199 yuan, but the knowledge base shows 1999 yuan, deducting 1 point." Completeness 5 points: covers three sub-issues: price, delivery time, and after-sales service. " This allows for precise defect localization during subsequent analysis.

Step Four: Analyze the results and iterate

After the evaluation is complete, my output is a CSV table, with each row showing samples + scores for each dimension + reasoning. Key points to focus on during analysis:

• Which dimensions generally score low: If the integrity average is only 2.5, it indicates the model frequently misses information.
• Which sample scores fluctuate greatly: Differences between evaluators or samples, often pointing to unclear standards.

My first run result: accuracy 4.2, completeness 3.0, tone 4.8. The reason for low integrity is that the model only answers the user's explicit questions without proactively providing related information (for example, when the user asks "When will the shipment be shipped?" the model answers the shipping time but does not specify the logistics inquiry method). So I added a prompt stating "If the user's issue involves orders, proactively add the order number to check the logistics." The second round of reviews improved completeness to 4.5.

The Most Prone Areas for Failure and Misunderstanding

The biggest pitfall of Playbooks is disconnect between datasets and standards. Many people write a few criteria based on templates, then randomly pull a dataset from HuggingFace to run reviews. The results look great but don't actually reflect business issues. The correct approach is: standards and datasets must come from the same business scenario, and small-scale validation should be done first.

Another common mistake is treating the review as a one-time acceptance. The value of Playbook lies in continuous regression: after every model update or prompt adjustment, it must be rerun, forming a cycle of "review - > analysis - > fix - > re-evaluation." If you only run once and then leave it there, then essentially, there is no review.

If you want to land now, what should you do first?

1. Choose a specific business scenario (such as customer service Q&A, code generation, summarization). Don't try to cover all abilities.
2. Define 2-3 actionable evaluation dimensions, clearly stating the deduction criteria for each dimension.
3. Collect 10 real samples and run them manually: You score using the standard yourself to see if the standards are clear and if the dataset is differentiated. This step takes less than an hour.
4. Adjust the criteria based on the results of manual runs, then expand to 50-100 samples.
5. Access automation frameworks (LangChain, DeepEval, etc.) and start the loop.

Where to go for systematic learning next

If you want to build a more complete evaluation system or learn how to systematically integrate Evals into Agent engineering, you can further read more systematic original paid articles andAIadvanced programming courses.