First hit the wall: There is an "invisible" context injection layer in the RAG pipeline
Three months ago, I took over a customer service assistant project based on a large model. The task itself is not complicated: retrieve relevant documents from the knowledge base, type in prompt and let the model answer. The early prototype ran smoothly, but it crashed once it was put into production - the model frequently ignored retrieved content and occasionally made up its own answers.
After two days of troubleshooting, I finally determined that the problem was not in retrieval, but in the prompt construction layer. There is a piece of logic in the code that directly inserts the retrieved text into the system message, but does not consider the token order, redundant information and role identification. As a result, the model gets a "noise pile" instead of clear context. This was the first time I seriously faced what Context Engineering was really about: it was not a simple splicing of prompts, but a dedicated engineering layer responsible for structuring, prioritization, and consistency.
Three conclusions that really changed the way I judge
1. The more contexts the better, but the more “aligned” the better
In the early days, I always wanted to put all the search results in, and was worried about missing them. However, the production logs show that when the context length exceeds the window that the model can effectively utilize (such as 4K tokens), the accuracy decreases. Later I switched to "relevance score + dynamic cropping": only keep the Top-3 clips, and enforce that each clip does not exceed 200 tokens. The results were immediate—answer accuracy increased from 68% to 91%.
2. The order of context injection directly affects the quality of reasoning
At first I put the knowledge base fragment at the end of the user message, thinking it would be the most straightforward. Results models often ignore the second half. After reviewing the literature, we found that the model is more sensitive to the beginning (primacy effect) and the end (recency effect). So I put the most critical facts at the beginning of the system message and the secondary information at the end of the user message. After adjustment, citations for key information jumped from 55% to 87%.
3. You must design "explicit boundaries" for the context
The model does not know which content is authoritative fact from the knowledge base and which is the user's own guess. In the past, I didn't add any tags, and as a result, the model often regarded misinformation casually said by users as fact. Later, I forced to add [来源：知识库] before each knowledge fragment, and made clear instructions in the system message: "Only use when the information marks the source, otherwise ignore." This seems simple, but it greatly reduces illusions.
Three pitfalls we have encountered and how to correct them
Pitfall 1: The context injection point is in the wrong position
Initially I put all the context processing logic in a function before the API call and did not separate it into a service. As a result, every time the injection strategy was adjusted, the business code had to be changed. The deployment cycle was three days, and iteration was extremely slow. Correction: Split Context Engineering into an independent middleware layer and inject strategies through configuration driver. After the change, the experimental iteration was shortened to hours.
Pitfall 2: Forgetting to handle context conflicts
In a test, the knowledge base returned two contradictory versions at the same time: "The price of product A is 100 yuan" and "The price of product A is 150 yuan (before discount)". The model randomly selected one, causing the user to confirm it repeatedly. Correction: Added conflict detection logic - if different values appear for the same entity, both versions will be automatically displayed, and the source and validity period will be marked. User complaints dropped by 80% after this.
Pitfall 3: Underestimating the impact of context caching
We used the context caching feature of the model API in hopes of reducing duplicate tokens. But I didn’t notice that the cache key design is too rough: as long as the system prompt remains unchanged, the cache will be reused. As a result, when the user's question is different, the cache still uses the old context, causing the answer to be incorrect. Correction: Add the digest hash of user query to the cache key. After that the cache hit rate dropped only 10% but the answer to the consistency question went back to zero.
The most worthy step to copy first: start with the log
If you also want to introduce Context Engineering, don’t write the strategy first, but create the log first. Specific methods:
- At the code that injects the context, output a "context audit log", which includes: the number of tokens at the time of injection, the source of each fragment, the priority score, and the injection location.
- Compare “what is injected” with “what is actually used in the model”. The simple method is to record the final generated prompt (including context) after each request, and then manually check it.
- Produce context quality reports weekly: which fragments are ignored, which are misquoted, and which lead to hallucinations.
Once you develop this habit, you can optimize your strategy data-drivenly rather than relying on guesswork. My team used logs for two weeks and found loopholes in all previous strategies.
When to persist and when to change routes
The effective boundary of Context Engineering lies in whether your system can stably obtain highly relevant source text. If the search recall rate is lower than 40% for a long time, revise the search first and do not try to make up for it with contextual methods. If the model frequently loses context in multiple rounds of dialogue, give priority to extended window models or plug-in memory modules instead of repeatedly carving out injection strategies.
In addition, if the team spends more than 50% of the time adjusting the context format every week but the effect is not obvious, it is recommended to stop and evaluate whether it should be switched to structured output constraints or fine-tuning the model. I have seen a case: a team changed the injection strategy five times in half a year, and the accuracy was always stuck at 75%. Later, they switched to using function calling to directly extract structured facts, and the accuracy jumped directly to 94%.
Next step of conversion direction
Context Engineering is just a link in the AI engineering system. If you want to transform from an ordinary developer to an engineer who can independently design and optimize Agent systems, we recommend reading the "Agent Engineering Practice" paid article series compiled by our team, which covers advanced topics such as context management, tool orchestration, and loop control in detail.
No comments yet. Be the first to share your thoughts.