EMNLP Findings2025
Inefficiencies of Meta Agents for Agent Design
Abstract
Recent works began to automate the design of agentic systems using meta-agents that propose and iteratively refine new agent architectures. In this paper, we examine three key challenges in a common class of meta-agents. First, we investigate how a meta-agent learns across iterations and find that simply expanding the context with all previous agents, as proposed by previous works, performs worse than ignoring prior designs entirely. We show that the performance improves with an evolutionary approach. Second, although the meta-agent designs multiple agents during training, it typically commits to a single agent at test time. We find that the designed agents have low behavioral diversity, limiting the potential for their complementary use. Third, we assess when automated design is economically viable. We find that only in a few cases-specifically, two datasets-the overall cost of designing and deploying the agents is lower than that of human-designed agents when deployed on over 15,000 examples. In contrast, the performance gains for other datasets do not justify the design cost, regardless of scale.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Self-Refine: Iterative Refinement with Self-Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution cites Self-Refine: Iterative Refinement with Self-Feedback
- Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Measuring Massive Multitask Language Understanding
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Self-Refine: Iterative Refinement with Self-Feedback
- Improving Factuality and Reasoning in Language Models through Multiagent Debate cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds on13
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Improving Factuality and Reasoning in Language Models through Multiagent DebateYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum et al.ICML 2024 · 1,562 citations
- Promptbreeder: Self-Referential Self-Improvement via Prompt EvolutionChrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero et al.ICML 2024 · 432 citations
Related papers
- MaMa: A Game-Theoretic Approach for Designing Safe Agentic SystemsJonathan Nöther, Adish Singla, Goran RadanovicICML 2026
- Automated Design of Agentic SystemsShengran Hu, Cong Lu, Jeff CluneICLR 2025
- MetaAgent: Automatically Constructing Multi-Agent Systems Based on Finite State MachinesYaolun Zhang, Xiaogeng Liu, Chaowei XiaoICML 2025
- Automated Stateful Specialization for Adaptive Agent SystemsMyan Vu, Harrish Ayyanar, Pang Jiang, Anwiketh Reddy et al.ICLR 2026
- Evolving Agentic Workflow Driven by Human-Agent CollaborationYuxin Liu, Jinxuan Zhang, Yuezhang Peng, Hefeng Zhou et al.ACL Findings 2026
