LLM Evals Workflow Checklist: Dismantling the core mechanism, boundaries and costs from an engineering perspective

Why do we need a LLM evals workflow checklist?

When many teams apply evals to LLM, the first reaction is "which framework to use" - LangSmith, MLflow, DeepEval or self-built. But after actually stepping into the pit, you will find that tool selection is only the last step. What really determines whether the eval pipeline is effective is: whether your problem definition is clear, whether the indicator selection matches the business scenario, and whether the data structure covers boundaries and failure modes.

This is an LLM evals workflow checklist for engineering teams. It is not a general guide, but is centered around the premise that "you have an LLM application and now you need to judge its quality."

Step 1: Problem Definition – What exactly are you trying to assess?

Scenario: A customer service summary model

Let’s say you’re evaluating an LLM that generates summaries for customer service conversations. The initial idea is to calculate ROUGE-L and BERTScore and compare them with manual annotation. It seemed reasonable, but after two weeks of actual operation, the team found that the ROUGE score was very high, but the business side complained that the summary often missed key decision points (such as "refunded" or "need to escalate the ticket").

This is a classic “disconnect between metrics and business goals” problem. ROUGE measures n-gram overlap, while the business needs factual completeness and decision traceability.

Check items:

• Is your evaluation goal to assist model iteration, or is it to serve as a barrier to entry for launch?
• If there are multiple business dimensions (factual accuracy, tone of voice, structural integrity), what metrics do you use to measure each?
• Do you distinguish between "model behavior" and "business results"? For example, if the abstract is concise enough but key information is missing, the two are quality issues at different levels.

Failure point: vague definition leads to indicator failure

The most common question is "Using semantic similarity to evaluate summary quality". Semantic Similarity can measure the consistency of meaning between two texts, but it cannot distinguish between "missing key facts" and "expressing different but semantically similar views." If the business requires "all key facts covered", then what you need is not similarity, but fact coverage or JSON-based structured validation.

Step 2: Operation steps - build a repeatable eval run

1. Construct evaluation data set

Don't just hand-write a dozen examples. One strategy: sample 200-500 real queries from production logs, and then manually mark "golden reference". If the cost of manual annotation is too high, you can use a small number of annotations + weak supervision to expand, but you must ensure that the test set is independent of the training set.

Check items:

• Does the test set cover normal scenarios, boundary scenarios (long text, rare words), and failure scenarios (incomplete user input, abnormal format)?
• Does each test input have a corresponding "expected output"? For open-ended tasks, there is no need for a single reference, but there must be "conditions that cannot be violated" (for example, "must not contain time information that does not appear in the user input").

2. Select and adapt indicators

Common combinations:

• Exact Match: Suitable for classification tasks.
• ROUGE/BLEU: suitable for text generation, but note that they are not sensitive to synonyms and sentence structure changes.
• LLM-based evaluation: Use GPT-4 or Claude to score the output. Clear prompts and scoring criteria need to be designed, otherwise LLM judges will introduce their own preferences.
• Custom rules: suitable for strong constraint scenarios, such as "The output must contain two JSON fields user_name and decision".

Check items:

• Are the main indicators you choose aligned with the business goals defined in the first step?
• Are "secondary indicators" added to detect side effects? For example, in addition to ROUGE, the summary task can also use "hallucination rate" as a one-vote veto indicator.

3. Operation and recording

Each eval run must be logged:

• Model version + parameters (temperature, top_p, etc.)
• Data batch identification
• Raw scores for all indicators
• Failure cases (outputs with extremely low scores or constraint violations)

If using LangSmith or MLflow, ensure that metrics naming and versions are traceable.

Failure point: only focus on the average score and ignore the distribution

An eval has an average score of 0.85, but it's actually 20% of the extremely bad examples that bring the overall score down? Or are 80% of the examples above 0.95? The avg may be the same in both cases, but the business impact is completely different. Check Item: When reporting metrics, P50, P90, P99, and the failure mode distribution must be output together.

Step 3: Misunderstanding - Four high-frequency pitfalls

Misunderstanding 1: Relying on a single indicator

Even if you use "ROUGE + BERTScore + GPT scoring" multiple indicators, if they are highly correlated (for example, ROUGE-L and ROUGE-1 are highly correlated), they actually do not provide much new information. You need to choose orthogonal metrics: one that measures factual accuracy, one that measures structure, and one that measures user preference.

Misunderstanding 2: The distribution difference between the test set and the training set is too large

Some teams split the test set directly from the training set, but the training set is often cleaned and formatted, and the input of the production environment is noisier. In the end, the eval result looked good, but it crashed after going online. Solution: Be sure to build an independent "production image test set" to simulate the real traffic distribution as much as possible.

Misunderstanding 3: Treat manual evaluation as the gold standard but do not perform consistency verification

If you rely on human evaluation (such as rating 3 annotators), be sure to calculate inter-annotator agreement (such as Cohen's Kappa). If it is only 0.3, it indicates that there is ambiguity in the task definition and the evaluation criteria need to be redesigned.

Misunderstanding 4: Ignoring the cost of eval itself

Each eval run has a computational cost and a time cost. If you run 100 test samples × 10 variants every day, the cost of calling the cloud API may exceed the cost of model fine-tuning. Checks: Regularly clean the unused eval cache and use a sampling strategy (such as only testing 10% of outlier samples) to balance coverage and cost.

Real scenario: a failed eval process

Team A built an eval pipeline for the customer service summary model, using ROUGE-L and manual scoring. Results: ROUGE-L 0.82, human rating 4.2/5. Two days after it went online, the customer complaint rate soared. Analysis found:

• Manual scoring samples are concentrated in "non-controversial" scenarios;
• ROUGE-L has insufficient weight on key entities such as "refunded";
• The test set does not cover the cases of "user contains swear words" or "customer service switches language".

Repair plan: Add 5% dirty data (including spelling errors, emoticons, and multi-language mixing) to the test set, and add a hard rule of "key entity coverage". If any summary misses the entity marked "must_cover", it will be directly judged as a failure.

Alternative: What if the eval pipeline doesn't work?

If you are just starting out, your computing power is limited, or your business scenarios change frequently, you can skip the full eval and switch to the three-step process of "log + random inspection + rule monitoring":

• Each output is recorded to the log, including input, output, and timestamp.
• 50 items are randomly selected every day, and an engineer quickly manually annotates whether they meet the basic requirements.
• Set several hard rules (such as JSON format verification, length limit, sensitive word detection), and trigger an alarm if they fail.

This lightweight solution can detect problems at a lower cost, and then build a complete eval pipeline until your business enters a stable period.

Summary

A good LLM evals workflow checklist should not just list the framework steps. The real value lies in helping you identify the goals, costs, and failure modes of each step. Start from the problem definition, choose indicators that are aligned with the business, build a test set that covers the boundaries, and then avoid common misunderstandings. If conditions do not permit, you can also start with a lightweight solution.