Lune

EMNLP Findings2022

Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models

Victor S. Bursztyn, David Demeter, Doug Downey, Larry Birnbaum

Abstract

How to usefully encode compositional task structure has long been a core challenge in AI. Recent work in chain of thought prompting has shown that for very large neural language models (LMs), explicitly demonstrating the inferential steps involved in a target task may improve performance over end-to-end learning that focuses on the target task alone. However, chain of thought prompting has significant limitations due to its dependency on huge pretrained LMs. In this work, we present compositional fine-tuning (CFT): an approach based on explicitly decomposing a target task into component tasks, and then fine-tuning smaller LMs on a curriculum of such component tasks. We apply CFT to recommendation tasks in two domains, world travel and local dining, as well as a previously studied inferential task (sports understanding). We show that CFT outperforms end-to-end learning even with equal amounts of data, and gets consistently better as more component tasks are modeled via fine-tuning. Compared with chain of thought prompting, CFT performs at least as well using LMs only 7.4% of the size, and is moreover applicable to task domains for which data are not available during pretraining.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization cites Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
  • DiNeR: A Large Realistic Dataset for Evaluating Compositional Generalization cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • STaR: Bootstrapping Reasoning With Reasoning cites Language Models are Few-Shot Learners
  • STaR: Bootstrapping Reasoning With Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event Extraction cites STaR: Bootstrapping Reasoning With Reasoning
  • CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs cites Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
  • CESAR: Automatic Induction of Compositional Instructions for Multi-turn Dialogs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts cites Language Models are Few-Shot Learners
  • AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts
  • Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey cites Language Models are Few-Shot Learners
  • Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A Survey cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Fundamental Capabilities of Large Language Models and their Applications in Domain Scenarios: A SurveyJiawei Li, Yizhe Yang, Yu Bai, Xiaofeng Zhou et al.ACL 2024 · 15 citations
  2. SLR: Automated Synthesis for Scalable Logical ReasoningLukas Helff, Ahmad Omar, Felix Friedrich, Antonia Wüst et al.ACL 2026 · 6 citations
  3. DiNeR: A Large Realistic Dataset for Evaluating Compositional GeneralizationChengang Hu, Xiao Liu, Yansong FengEMNLP 2023
  4. Reflective Agreement: Combining Self-Mixture of Agents with a Sequence Tagger for Robust Event ExtractionFatemeh Haji, Mazal Bethany, Cho-Yu Jason Chiang, Anthony Rios et al.EMNLP 2025
  5. CESAR: Automatic Induction of Compositional Instructions for Multi-turn DialogsTaha Aksu, Devamanyu Hazarika, Shikib Mehri, Seokhwan Kim et al.EMNLP 2023

Builds on15

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
  4. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model PromptsTongshuang Wu, Michael Terry, Carrie Jun CaiCHI 2022 · 465 citations
  5. Measuring Compositional Generalization: A Comprehensive Method on Realistic DataDaniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman et al.ICLR 2020 · 401 citations

Related papers

  1. Chain-of-Instructions: Compositional Instruction Tuning on Large Language ModelsShirley Anugrah Hayati, Taehee Jung, Tristan Bodding-Long, Sudipta Kar et al.AAAI 2025 · 14 citations
  2. Chain-of-Thought Tuning: Masked Language Models can also Think Step By Step in Natural Language UnderstandingCaoyun Fan, Jidong Tian, Yitian Li, Wenqing Chen et al.EMNLP 2023 · 3 citations
  3. Decomposed Prompting: A Modular Approach for Solving Complex TasksTushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu et al.ICLR 2023 · 94 citations
  4. Compositional Semantic Parsing with Large Language ModelsAndrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales et al.ICLR 2023 · 39 citations
  5. Teaching LLMs How to Learn with Contextual Fine-TuningYounwoo Choi, Muhammad Adil Asif, Ziwen Han, John Willes et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines