Lune

ACL Findings2024

Small Models are Valuable Plug-ins for Large Language Models

Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, Julian J. McAuley

Abstract

Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their weights are often publicly unavailable and their immense sizes make the models difficult to be tuned with common hardware. As a result, effectively tuning these models with large-scale supervised data can be challenging. As an alternative, In-Context Learning (ICL) can only use a small number of supervised examples due to context length limits. In this paper, we propose Super In-Context Learning (SuperICL) which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks. Our experiments demonstrate that SuperICL can improve performance beyond state-of-the-art fine-tuned models while addressing the instability problem of in-context learning. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

31 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Supervised Knowledge Makes Large Language Models Better In-context Learners cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
  • Supervised Knowledge Makes Large Language Models Better In-context Learners cites An Explanation of In-context Learning as Implicit Bayesian Inference
  • Supervised Knowledge Makes Large Language Models Better In-context Learners cites Language Models are Few-Shot Learners
  • PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning cites An Explanation of In-context Learning as Implicit Bayesian Inference
  • A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis cites Language Models are Few-Shot Learners
  • Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Language Models are Few-Shot Learners
  • An Explanation of In-context Learning as Implicit Bayesian Inference cites Language Models are Few-Shot Learners
  • Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents cites HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
  • LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty cites Language Models are Few-Shot Learners
  • LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty cites HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
  • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites An Explanation of In-context Learning as Implicit Bayesian Inference
  • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites Toolformer: Language Models Can Teach Themselves to Use Tools
  • HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers31

  1. A Real-World WebAgent with Planning, Long Context Understanding, and Program SynthesisIzzeddin Gur, Hiroki Furuta, Austin V. Huang, Mustafa Safdari et al.ICLR 2024 · 359 citations
  2. Plug-and-Play Policy Planner for Large Language Model Powered Dialogue AgentsYang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng et al.ICLR 2024 · 86 citations
  3. LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using UncertaintyZhen Zhang, Yuhua Zhao, Hang Gao, Mengting HuWWW 2024 · 50 citations
  4. PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context LearningHyeong Kyu Choi, Yixuan LiICML 2024 · 31 citations
  5. Supervised Knowledge Makes Large Language Models Better In-context LearnersLinyi Yang, Shuibai Zhang, Zhuohao Yu, Guangsheng Bao et al.ICLR 2024 · 28 citations

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
  3. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li et al.NeurIPS 2023 · 1,778 citations
  4. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
  5. An Explanation of In-context Learning as Implicit Bayesian InferenceSang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu MaICLR 2022 · 1,030 citations

Related papers

  1. Task Descriptors Help Transformers Learn Linear Models In-ContextRuomin Huang, Rong GeICLR 2025
  2. Generative Calibration for In-context LearningZhongtao Jiang, Yuanzhe Zhang, Cao Liu, Jun Zhao et al.EMNLP Findings 2023 · 4 citations
  3. What Do Language Models Learn in Context? The Structured Task HypothesisJiaoda Li, Yifan Hou, Mrinmaya Sachan, Ryan CotterellACL 2024 · 5 citations
  4. Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity RecognitionQi Zhang, Fangping Lan, Cornelia Caragea, Longin Jan Latecki et al.ACL Findings 2026
  5. In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space SteeringSheng Liu, Haotian Ye, Lei Xing, James Y. ZouICML 2024 · 244 citations
Dusk over the sea between two cliffs drawn in fine vertical lines