ACL Findings2025
Inductive Linguistic Reasoning with Large Language Models
Abstract
Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models' knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as +8.1% and Llama-3.1-405B-Instruct by +5.9% over chain-of-thought approaches. Furthermore, we demonstrate that our method generalizes to other tasks present in Linguistics Olympiad competitions, achieving state-of-theart results across nearly all problem types and difficulty levels in the LINGOLY dataset.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models cites Language Models are Few-Shot Learners
- Graph of Thoughts: Solving Elaborate Problems with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers1
Builds on15
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
- Graph of Thoughts: Solving Elaborate Problems with Large Language ModelsMaciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger et al.AAAI 2024 · 1,292 citations
Related papers
- Large Language Models as Analogical ReasonersMichihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat et al.ICLR 2024 · 155 citations
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought PromptingHaoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao et al.EMNLP Findings 2023 · 76 citations
- LingGym: How Far Are LLMs from Thinking Like Field Linguists?Changbing Yang, Franklin Ma, Freda Shi, Jian ZhuEMNLP 2025
- Can Language Models Serve as Analogy Annotators?Xiaojing Zhang, Bochen LyuACL Findings 2025
- ThinkSum: Probabilistic reasoning over sets using large language modelsBatu Ozturkler, Nikolay Malkin, Zhen Wang, Nebojsa JojicACL 2023 · 10 citations
