ACL Findings2024
Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers
Tuo Zhang, Jinyue Yuan, Salman Avestimehr
Abstract
Numerous recent works aim to enhance the efficacy of Large Language Models (LLMs) through strategic prompting. In particular, the Optimization by PROmpting (OPRO) approach provides state-of-the-art performance by leveraging LLMs as optimizers where the optimization task is to find instructions that maximize the task accuracy (Yang et al., 2023). In this paper, we revisit OPRO for automated prompting with relatively small-scale LLMs, such as LLaMa-2 family and Mistral 7B. Our investigation reveals that OPRO shows limited effectiveness in small-scale LLMs, with limited inference capabilities constraining optimization ability. We suggest future automatic prompting engineering to consider both model capabilities and computational costs. Additionally, for small-scale LLMs, we recommend direct instructions that clearly outline objectives and methodologies as robust prompt baselines, ensuring efficient and effective prompt engineering in ongoing research.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond cites Large Language Models are Zero-Shot Reasoners
- Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models Cannot Self-Correct Reasoning Yet cites Large Language Models are Zero-Shot Reasoners
- Large Language Models Cannot Self-Correct Reasoning Yet cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers cites Large Language Models are Zero-Shot Reasoners
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers cites Language Models are Few-Shot Learners
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers2
- Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and BeyondDingzirui Wang, Xuanliang Zhang, Keyan Xu, Qingfu Zhu et al.ICLR 2026 · 3 citations
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt OptimizersSarkar Snigdha Sarathi Das, Ryo Kamoi, Bo Pang, Yusen Zhang et al.ICLR 2025
Builds on11
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
- Large Language Models Cannot Self-Correct Reasoning YetJie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng et al.ICLR 2024 · 858 citations
Related papers
- Large Language Models as OptimizersChengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu et al.ICLR 2024 · 817 citations
- Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model OptimizersXinyu Tang, Xiaolei Wang, Wayne Xin Zhao, Siyuan Lu et al.AAAI 2025 · 36 citations
- ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model CapabilitiesYifan Duan, Yihong Tang, Kehai Chen, Liqiang Nie et al.EMNLP 2025
- StraGo: Harnessing Strategic Guidance for Prompt OptimizationYurong Wu, Yan Gao, Bin Zhu, Zineng Zhou et al.EMNLP Findings 2024 · 5 citations
- MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt OptimizationYuyan Chen, Zhihao Wen, Ge Fan, Zhengyu Chen et al.EMNLP Findings 2023 · 18 citations
