ACL Findings2023
Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning
Zhanming Jie, Wei Lu
Abstract
Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language processing tasks, but designing prompts that generalize well to diverse problem types can be challenging (Zhou et al., 2022), especially in the context of math word problem (MWP) solving. Additionally, it is common to have a large amount of training data that have a better diversity coverage but CoT annotations are not available, which limits the use of supervised learning techniques. To address these issues, we investigate two approaches to leverage the training data in a few-shot prompting scenario: dynamic program prompting and program distillation. Our approach is largely inspired by Gao et al. (2022), where they proposed to replace the CoT with the programs as the intermediate reasoning step. Such a prompting strategy allows us to accurately verify the answer correctness through program execution in MWP solving. Our dynamic program prompting involves annotating the training data by sampling correct programs from a large language model, while program distillation involves adapting a smaller model to the programannotated training data. Our experiments on three standard MWP datasets demonstrate the effectiveness of these approaches, yielding significant improvements over previous baselines for prompting and fine-tuning. Our results suggest that leveraging a large amount of training data can improve the generalization ability of prompts and boost the performance of finetuned small models in MWP solving 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- STaR: Bootstrapping Reasoning With Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- ReFT: Reasoning with Reinforced Fine-Tuning cites PAL: Program-aided Language Models
- ReFT: Reasoning with Reinforced Fine-Tuning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- ReFT: Reasoning with Reinforced Fine-Tuning cites Training language models to follow instructions with human feedback
- PAL: Program-aided Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Exploring Equation as a Better Intermediate Meaning Representation for Numerical Reasoning of Large Language Models cites PAL: Program-aided Language Models
- Exploring Equation as a Better Intermediate Meaning Representation for Numerical Reasoning of Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers2
Builds on19
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
- PAL: Program-aided Language ModelsLuyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon et al.ICML 2023 · 700 citations
Related papers
- Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language ModelsLei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu et al.ACL 2023 · 249 citations
- Towards Revealing the Mystery behind Chain of Thought: A Theoretical PerspectiveGuhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye et al.NeurIPS 2023 · 470 citations
- Plan, Verify and Switch: Integrated Reasoning with Diverse X-of-ThoughtsTengxiao Liu, Qipeng Guo, Yuqing Yang, Xiangkun Hu et al.EMNLP 2023 · 7 citations
- Fewer is More: Boosting Math Reasoning with Reinforced Context PruningXijie Huang, Li Lyna Zhang, Kwang-Ting Cheng, Fan Yang et al.EMNLP 2024 · 7 citations
- CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts ReasoningYuanheng Fang, Guoqing Chao, Wenqiang Lei, Shaobo Li et al.AAAI 2025 · 4 citations
