Task Descriptors Help Transformers Learn Linear Models In-Context

Ruomin Huang; Rong Ge

ICLR2025Top-tier venue

Task Descriptors Help Transformers Learn Linear Models In-Context

Ruomin Huang, Rong Ge

Publisher

2025Year

1Top-tier citations

Abstract

Large language models (LLM) exhibit strong in-context learning (ICL) ability, which allows the model to make predictions on new examples based on the given prompt. Recently, a line of research (

Ask about this paper

Your agent reads all of it.

Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.

What it claims
Who builds on it
How it compares
Where it falls short

You ask

Your agent calls

Luneget_paper_fulltext

Ask in Lune

Free to start. No credit card required.

Terminal

Install the CLI

$ lune papers fulltext 69f28194-4614-49aa-9e81-82750ec5b7ac

Cited by top-tier papers1

Provable In-Context Vector Arithmetic via Retrieving Task ConceptsDake Bu, Wei Huang, Andi Han, Atsushi Nitanda et al.ICML 2025

Ask how each one uses it

Builds on7

Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
What Can Transformers Learn In-Context? A Case Study of Simple Function ClassesShivam Garg, Dimitris Tsipras, Percy Liang, Gregory ValiantNeurIPS 2022 · 883 citations
Transformers Learn In-Context by Gradient DescentJohannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento et al.ICML 2023 · 729 citations
Transformers learn to implement preconditioned gradient descent for in-context learningKwangjun Ahn, Xiang Cheng, Hadi Daneshmand, Suvrit SraNeurIPS 2023 · 324 citations
One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-AttentionArvind V. Mahankali, Tatsunori Hashimoto, Tengyu MaICLR 2024 · 160 citations

Abstract

Ask about this paper

Your agent reads all of it.

Cited by top-tier papers1

Builds on7

Related papers