Skip to content
AIMUG
← All talks

Wed, Mar 4, 202621:08EvalsRAGCoding agents

Optimizing AI Systems: A Coding Agent in a Loop with CodeEvolver

Share this talkLinkedInX

Julian Ghadially

Julian Ghadially puts a coding agent in a loop that fixes AI system failures by changing prompts and code.

Watch on YouTubeRecorded at AIMUG, Mar 4, 2026

Takeaways

  • Prompt optimization lets an optimizer write prompts from a dataset and reward function; code optimization also lets it change the code, such as adding sub-agents, tools or context steps.
  • GEPA (Genetic Pareto) is the prompt optimizer behind it; CodeEvolver, his open-source tool, runs Claude Code in a sandbox that makes Git branches and keeps the winners against an eval set.
  • He reports 5 to 15 points over prompt optimization on HoVer and HotpotQA, measured on a held-out test set with a resource penalty and controlled for chain-of-thought reasoning.
  • The best HotpotQA trajectory was short: two-hop retrieval, summarization dropped, answer extraction instead of generation, and 10 pages per search instead of 7.
  • Q&A covers Claude Code in a sandbox, agile and Six Sigma views, heavy versus light mode, and optimizer cost of about $0.50 to $1 per iteration and $40 to $60 per cycle, not counting running the system itself.
SpeakerJulian GhadiallyCo-founder, CodeEvolverAll talks →

Transcript

Transcript coming.

Same night

Related talks