이 문제가 처음 발생한 실제 시나리오는 무엇입니까?
At the beginning of the year, I was working on an automated code review tool. The goal was to allow the Agent to automatically repair the code and resubmit it based on test failure information. The initial solution was very straightforward: the Agent calls LLM once, gets repair suggestions, applies patches, and runs tests. But problems were quickly discovered—the fixes generated for the first time were often incorrect, or even introduced new bugs.
At this time, I really hit Loop Engineering: instead of simply "calling LLM to get the results", you need to design a loop-evaluate the output, feedback correction, and re-evaluate until it passes or reaches the termination condition. Many articles talk about theoretical "closed loops", but what I face is the actual retry queue, termination condition judgment and status management.
이 일을 하고 나서 내 판단을 완전히 바꿔놓은 몇 가지 결론
1. The more cycles the better, the key is signal quality.
I initially set max_retries=5, thinking that I would be able to pass it if I tried a few more times. In fact, after the third time, Agent often only fine-tunes punctuation or changes synonyms, without any substantial improvement. Later, I changed the termination condition from "fixed number of times" to "signal change rate" - if the semantic similarity of two consecutive repair solutions exceeds 90%, it will be terminated directly and marked as "cannot be automatically repaired".
2. Context inflation is the invisible killer
Each loop sends the complete conversation history to the model, and the context window quickly fills up, and the model begins to "forget" the original code. After I stepped on this pitfall, I changed to retaining only the current code block, the latest failure information and the previous repair diff. As a result, the repair success rate increased by 12%.
3. The reliability of the test determines the effectiveness of the cycle
If the test itself has flaky factors, such as network jitter or timing issues, the loop will try again and again on the wrong signal. It took me two weeks to get the flaky test ratio from 8% to under 1% before the loop was really instructive.
당시 우리가 겪었던 함정과 이후 이를 어떻게 수정했는지
Pit 1: There is no distinction between "retryable errors" and "non-retryable errors"
Initially retry all failures, including compilation errors (such as missing imports) - retrying these 10 times will not get better and will only waste time and tokens. The correction method is to pre-define error categories: syntax errors, missing dependencies, logic errors - the first two categories directly report errors and terminate, and only "logic that is not perfect" can enter the loop.
Pit 2: The circular log is too thick and cannot be debugged
Initially, only "retry 1: fail, retry 2: fail" is recorded, and the cause of the failure cannot be seen at all. Later I printed out at each loop node: original error, model output diff, test output summary. With this log, the time to locate the problem is reduced from hours to minutes.
Pit 3: Failure to consider the cost limit
A complex repair job might run 8 cycles and cost $2. Not sustainable for the experimental phase. My correction plan is: set a token budget, and once the accumulated tokens in the current cycle exceed the expected value of the entry (such as the estimated labor cost of fixing this bug), it will automatically downgrade to simple LLM or suspend it directly.
비슷한 작업을 하고 있다면 가장 먼저 복사해야 할 가장 가치 있는 단계
Add this check (pseudocode) directly to the entry of your repair loop:
def should_retry(error, history):
if error.type in UNRECOVERABLE_ERRORS:
return False
if len(history) >= 2 and similarity(history[-1], history[-2]) > 0.9:
return False
if total_tokens_used > BUDGET:
return False
return True
These three-point checks cover the most common problems: unrepairable errors, models spinning in circles, and out-of-control costs. Putting it in the outermost layer of the loop can save at least 30% of unnecessary consumption.
언제 투자를 계속해야 하며 언제 노선을 바꿔야 할까요?
Signal to continue investing:
- You have clear signals (e.g. test pass rate, actual fix rate) that you are continuing to improve, even if it is slow.
- You have added the above termination and degradation logic to the loop design, and the cost is controllable.
- The team has the energy to maintain test stability and bug classification lists.
Signal to change route:
- You find that most of the failures are the same kind of "non-retryable" errors, such as dependency version conflicts or API changes - this is the time to fix the source problem, not the loop.
- The number of cycles has increased, but the repair success rate has been chronically below 30%. This may mean that LLM itself is not good at this task and needs to change the model or change the prompt strategy.
- You find yourself spending 80% of your time adjusting loop parameters (max_retries, temperatures, top_p) instead of improving the task itself.
When I encountered these signals, I decisively suspended the automatic repair loop and changed to a semi-automatic solution: the loop is only responsible for generating suggestions, and then implements them after manual confirmation. Two months later, I waited for the model version to be upgraded before trying full automation again.
요약
Loop Engineering is not a silver bullet, it solves tasks with "clear feedback signals, room for retry, and acceptable cost". If these conditions are not met in your scenario, don't be rigid. But if you happen to face this kind of problem, the above experience-especially termination condition design, context management, error classification and budget control-can be directly used.
Next, if you want to get a more systematic Agent engineering methodology, including context window strategies, cycle cost models and error recovery models, you can enter our paid articles and courses.
아직 댓글이 없습니다