Tech • AI • Robotics • Game
Tool, skill, or subagent? Decomposing an agent that outgrew its prompt
6/10TL;DR
Refactoring an overgrown AI agent by simplifying prompts, reducing tools, and restructuring sub-agents improved performance from 83% to 92% while cutting cost and latency.
KEY POINTS
Agent complexity leads to performance decay
An inventory management system called Stock Pilot accumulated features over time, expanding to a 400-line system prompt, 12 tools, and multiple sub-agents. While each addition solved a business need, the lack of architectural updates led to regressions, inefficiencies, and declining evaluation scores.
Evaluation framework exposed systemic issues
The system was tested across 12 evaluation tasks, including regression and multi-turn failure scenarios. Initial performance dropped as low as 62% pass rate, with failures tied to inefficiency, conflicting instructions, and breakdowns between orchestrators and sub-agents.
Context overload caused reasoning errors
A key failure in forecasting showed the agent retrieving correct inputs but applying the wrong multiplier, highlighting context confusion rather than model weakness. Overloaded and contradictory instructions in the system prompt led to hallucination-like behavior.
System prompt reduced via “skills” abstraction
Engineers replaced the long system prompt with a ~15–50 line core prompt, moving conditional knowledge into skills—modular, on-demand context blocks. This “progressive disclosure” approach ensured the model accessed only relevant information per task, improving clarity and efficiency.
Tooling simplified to core primitives
The original 12 tools were largely removed and replaced with basic primitives such as code execution, file system access, and web search. This reduced token usage dramatically, from over 200,000 tokens per task to far lower levels, while also lowering cost and execution time.
Sub-agent usage narrowed and clarified
Multiple sub-agents were eliminated due to coordination issues. A single forecasting sub-agent was retained to provide an isolated reasoning environment, reflecting two valid use cases: parallel problem-solving and independent review contexts.
Communication failures identified as key risk
One evaluation failed despite correct logic because of misalignment between orchestrator and sub-agent outputs. This underscored the difficulty of maintaining consistency across distributed agent components.
Shift to managed infrastructure improved scalability
Migrating to Claude Managed Agents removed the burden of infrastructure concerns such as scaling, security, and session management. This allowed developers to focus exclusively on agent design and behavior.
Token efficiency directly reduced costs
By enabling code execution instead of embedding large datasets in context, the system reduced both token consumption and operational cost, while maintaining or improving output quality.
Iterative “hill climbing” with evals drove improvement
Continuous evaluation and incremental fixes—referred to as “hill climbing”—enabled steady performance gains. The final system achieved approximately 92% success rate, with better consistency and lower resource usage.
CONCLUSION
Streamlining agent architecture through modular context, simpler tools, and disciplined evaluation significantly improves reliability, cost efficiency, and scalability in complex AI systems.
Explain this