5 engineering lessons I learned after Harnesses exploded

1. In which real-life scenario did I first encounter this problem?

It happened while writing unit tests for a multi-step Agent workflow. The Agent needs to call the MCP tool in sequence, read the context and make conditional jumps. Conventional pytest mock replaces each tool call with a fixed return value, but the test always passes, but times out frequently after going online. After procrastinating for two days, I discovered that the problem was that the mock did not simulate the delay and error response of the real tool - and harness is the tool used to solve the "difference between the test environment and the production environment".

Harnesses (test harness) are essentially a lightweight execution framework that takes over the input and output of the Agent, allowing you to inject mock data, simulate errors, and record behaviors in an isolated environment. But the first pitfall I stepped into at that time was: using the harness as an ordinary mock, only replacing the return value, completely ignoring the timeout, retry and exception flow.

2. Several conclusions that really changed my way of judgment after doing this.

After successfully stabilizing the test with the harness, I came to three key conclusions:

• Harness is not a universal glue. It is good at simulating external dependencies, but it cannot replace the code logic itself. If there is a bug in your Agent's decision-making logic, harnessing will only expose the bug faster - which is a good thing, but many teams mistakenly believe that harnessing can make vulnerable code pass tests.
• The value of Harness lies in "controllable uncertainty". Agent behavior often relies on the random output of LLM. Harness can fix prompts and tool outputs, making each test behavior repeatable. This is more thorough than simply mocking the LLM endpoint because it also captures the context assembly process.
• The learning curve is steeper than you think. Harnesses have many configuration items (timeout, retry, error injection, logging), and it is easy to introduce new bugs during the configuration phase. It initially took me three days to get the first harness running and just one day to write the actual tests.

3. The pitfalls we encountered at that time and how we corrected them later

The biggest pitfall is excessive harness. At the time I wanted to stuff all external calls into the harness, including a static function that just returned a constant version. As a result, the harness object becomes bloated and a bunch of unnecessary mocks are loaded for each test. The fix is ​​simple: only manage truly external unstable dependencies (such as LLM calls, third-party APIs, file systems) in the harness, and cover pure computational logic with normal unit tests.

The second pitfall is ignoring the life cycle of the harness. Harnesses typically have setup and teardown phases, and I initially failed to clean up the context between tests, resulting in multiple tests sharing the same mock state and downstream tests being contaminated by upstream side effects. It was later solved by explicitly calling reset() before and after each test.

The third pitfall is incomplete error simulation. I only simulated normal returns and timeouts, but missed some errors (e.g. authentication failure, rate limiting). As a result, the Agent encountered a 429 error after going online, and the harness test never covered this situation. The lesson is: when simulating errors with harness, at least cover the three common exceptions: timeout, HTTP 4xx/5xx, and network outages.

4. If you are doing similar work, the most worthwhile step to copy first

The most direct starting action: Use harness to write a "template harness" for the most critical external call in your Agent (such as the only LLM API). The specific operations are:

1. Select the core unstable dependency in the current project (usually LLM or database).
2. Write a harness class, including setup (initializing the mock server), execute (calling the Agent and capturing the output), and teardown (cleaning the mock).
3. Fix the response of the call in the harness (for example, return a fixed JSON), and record the prompt and parameters that were actually called.
4. Run a test to confirm that the output is as expected.

This step will allow you to get the first reusable harness test within half an hour, and immediately find out whether the prompt is spliced ​​correctly and whether the parameters are passed correctly. My experience is that 90% of prompt errors can be exposed by running the harness once.

5. When should you continue investing and when should you change your route?

Continue to invest in the harness scenario:

• Your Agent has more than 3 external dependencies.
• You are developing new functionality and need to ensure that the new code does not break existing behavior.
• The team is constantly adding new agents or new tools.

Signal that it’s time to change route:

• You spent more than 2 days still debugging the harness configuration itself instead of writing business tests.
• Your external dependencies change frequently, causing harness maintenance work to outweigh testing benefits.
• The project is still in the prototype stage, and the API and process are changing every day.

In these cases, use a lighter mock to support it first, and then introduce the harness after the stability is improved.

Write at the end

Harnesses are not silver bullets, but used in the right place can greatly improve the testing confidence of Agent projects. The key is to clarify its applicable boundaries: control non-determinism, simulate real failures, and isolate external dependencies. Hopefully these 5 lessons will help you avoid some detours.