Dismantling Evals For AI Agents from an engineering perspective: core mechanism, boundaries and costs

Why agent evaluation is harder than model evaluation

When your AI program is no longer a single question and answer, but an Agent that can call tools, maintain context, and multi-step reasoning, traditional accuracy indicators become ineffective. An agent may perform perfectly in a single conversation, but gradually deviate from the goal over 10 consecutive rounds of interaction - such as forgetting previous user preferences, or calling the API multiple times by mistake.

This is why Evals are crucial to the AI Agent. It is no longer simply comparing output tokens, but requires designing multi-dimensional tasks that can reflect "decision quality" and "behavior coherence".

Core Mechanism: What exactly is Evals testing?

A mature Agent Evals system usually covers three levels:

1. Task Completion Rate: Did the Agent achieve the user goal within the specified round? For example, "help the user book an Italian restaurant at 7 o'clock tomorrow night". The success criteria include: finding the restaurant, confirming the time, and issuing a reservation request. The detail that is easily overlooked here is that if the Agent tries 5 times before succeeding, is it considered completed? In actual deployment, many failures come from improper retry limit and timeout handling.

2. Tool Call Compliance: Does the Agent call the correct tool and call it with the correct parameters? For example, a weather query agent should call the weather API instead of searching the encyclopedia. Tool calling errors are the number one cause of Agent failures. Especially when the number of tools exceeds 5, the internal routing logic is easily confused.

3. Security & Boundary Constraints: Will the Agent perform dangerous operations? For example, deleting user data and sending unconfirmed emails. This part of the assessment often relies on a combination of rules engines and manual review, because fully automated security assessments inherently carry the risk of false negatives.

Real scenario failure review: an Agent evaluates the missed boundary

Suppose you design an evaluation data set for a customer service agent. You have prepared 100 test cases, covering common scenarios such as returns, refunds, and logistics inquiries. After passing the evaluation, the Agent goes online.

In the first week, a user inquired, "The product I bought is fake, can I get a refund?" - Agent directly called the refund API, without requiring the user to upload evidence, and without triggering manual review. This is a typical evaluation under-coverage:

• Reason for Failure: The evaluation dataset only contains "Standard Return Process" and does not include the variants "Disputed Returns" or "Potential for Fraud".
• Boundary Problem: The Agent's decision-making logic is "if the user mentions a refund, call the refund API", which lacks classification and risk classification of reasons.
• Executable Practice: When designing evaluation use cases, negative examples and disturbances must be included - such as users expressing dissatisfaction but not explicitly requesting a refund, or users requesting a refund but the product has been used. At the same time, the assessment report should mark "boundary scenarios not covered" rather than just giving a "pass/fail" score.

The most likely place to fail and misunderstanding

1. Assess task contamination (Data Contamination)

If you use public agent benchmark data sets to evaluate your agent, you are likely to get inflated scores. Because the Agent model may have seen these tasks in the pre-training stage.

Judgment basis: Check whether the release time of the benchmark data set is later than your model training deadline. A safer approach would be to build your own private evaluation task, or at least perform random perturbations on a public dataset.

2. Treat success rate as the only indicator

Many teams only focus on "whether the task is completed" and ignore the costs in the completion process - such as the number of API calls, inference time, and the number of corrections required by the user. An Agent that has a 90% success rate but takes an average of 3 minutes may not be as effective in actual business as an Agent that has an 80% success rate but takes 10 seconds to complete.

Executable practices: Establish a multi-metric dashboard, including at least: success rate, average call rounds, average inference time, and user intervention rate.

3. Ignoring the cost of assessment itself

Each evaluation calls LLM inference one or more times. If you have 1,000 evaluation use cases every day, with an average of 5 steps of reasoning per use case, the total token consumption may exceed the budget. More critically, the evaluation results may be unstable due to fluctuations in the model service load.

Cost Boundary: It is necessary to preset an "evaluation budget upper limit" and review the ROI of the evaluation regularly. Sometimes using a small model (such as GPT-4o-mini) for rapid screening and then using a large model for accurate judgment can effectively reduce costs.

If you want to land now, what should you do as the first step?

Step chain: Build an evaluation board from scratch

1. Define Core Behaviors (1-2 days) List 3-5 key behaviors that your Agent must do right. For example: the shopping agent must be able to correctly list the products in the shopping cart, modify the quantity, and check out. Don't try to cover all scenarios at once.

2. Build 10-20 basic test cases (1 day) Each behavior corresponds to at least 2 use cases: one normal path and one edge case. For example, for "modify quantity": the normal value is changed to a positive number, and the boundary is changed to 0 (should trigger confirmation) or changed to a negative number (should report an error and maintain the original quantity).

3. Run the first assessment (half day) Record the pass/fail of each use case and the Agent’s reasoning trace. The core goal of this step is to discover "simply unfeasible" scenarios, rather than chasing high scores.

4. Iteratively expand the use case library (continuous) Supplement 5-10 new use cases every week based on real failure cases from production logs. Priority: Handle high-risk cases that result in data loss or financial loss first.

5. Establish a return mechanism (monthly) After each Agent update, run the complete evaluation suite. There must be an automated script that can output a comparison report within 30 minutes.

Guide to avoid pitfalls

• Don’t pursue automated assessments right from the start. It is much more effective to manually annotate 100 use cases and review the traces one by one than to run 10,000 automated use cases without anyone reviewing them.
• Pay attention to the ambiguity of evaluation terms: For example, what are the evaluation criteria for "finding related products"? Does it include keywords, price range matching, or did the user click on the recommendation? Actionable definitions must be clear.

Where to go next to continue systematic learning

To transform from an ordinary developer to an agent engineer, evaluation is only one part of the process. You need to systematically master: decision loop (Loop) design, context window management, tool definition and invocation, model selection and cost optimization. This knowledge is difficult to put together into a complete puzzle in fragmented blogs.

If you have completed the first step of building the evaluation board above, it means that you have a practical foundation. The next focus is: How ​​to polish your Agent from a "prototype" to a "product-level" system - including robustness testing, long-range task decomposition, failure recovery strategies, etc.

We will gradually dismantle these modules in high-quality original paid articles and AI advanced programming courses. Each article is an independently executable engineering practice rather than a theoretical exposition. You can choose the corresponding chapter to start based on your current project needs.