Wed, Mar 4, 202621:08EvalsRAGCoding agents
Optimizing AI Systems: A Coding Agent in a Loop with CodeEvolver
Julian Ghadially puts a coding agent in a loop that fixes AI system failures by changing prompts and code.
Takeaways
- Prompt optimization lets an optimizer write prompts from a dataset and reward function; code optimization also lets it change the code, such as adding sub-agents, tools or context steps.
- GEPA (Genetic Pareto) is the prompt optimizer behind it; CodeEvolver, his open-source tool, runs Claude Code in a sandbox that makes Git branches and keeps the winners against an eval set.
- He reports 5 to 15 points over prompt optimization on HoVer and HotpotQA, measured on a held-out test set with a resource penalty and controlled for chain-of-thought reasoning.
- The best HotpotQA trajectory was short: two-hop retrieval, summarization dropped, answer extraction instead of generation, and 10 pages per search instead of 7.
- Q&A covers Claude Code in a sandbox, agile and Six Sigma views, heavy versus light mode, and optimizer cost of about $0.50 to $1 per iteration and $40 to $60 per cycle, not counting running the system itself.

Transcript
Transcript coming.



