Density estimation with LLMs: a geometric investigation of in-context learning trajectories
Toni J. B. Liu, Nicolas Boullé, Raphaël Sarfati, Christopher J. Earls
Abstract
Large language models (LLMs) demonstrate remarkable emergent abilities to perform in-context learning across various tasks, including time-series forecasting. This work investigates LLMs' ability to estimate probability density functions (PDFs) from data observed in-context; such density estimation (DE) is a fundamental task underlying many probabilistic modeling problems. We use Intensive Principal Component Analysis (InPCA) to visualize and analyze the in-context learning dynamics of LLaMA-2, Gemma, and Mistral. Our main finding is that these LLMs all follow similar learning trajectories in a low-dimensional InPCA space, which are distinct from those of traditional density estimation methods such as histograms and Gaussian kernel density estimation (KDE). We interpret these LLMs' in-context DE process as a KDE algorithm with adaptive kernel width and shape. This custom kernel model captures a significant portion of the LLMs' behavior despite having only two parameters. We further speculate on why the LLMs' kernel width and shape differ from classical algorithms, providing insights into the mechanism of in-context probabilistic reasoning in LLMs. Our codebase, along with a 3D visualization of an LLM's in-context learning trajectory, is publicly available at https://github.com/AntonioLiu97/LLMICL_inPCA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixtures cites An Explanation of In-context Learning as Implicit Bayesian Inference
- Transformers as Unsupervised Learning Algorithms: A study on Gaussian Mixtures cites Language Models are Few-Shot Learners
- Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-context cites An Explanation of In-context Learning as Implicit Bayesian Inference
- Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-context cites Language Models are Few-Shot Learners
- Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-context cites Large Language Models Are Zero-Shot Time Series Forecasters
- An Explanation of In-context Learning as Implicit Bayesian Inference cites Language Models are Few-Shot Learners
- Large Language Models Are Zero-Shot Time Series Forecasters cites Language Models are Few-Shot Learners
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models cites Language Models are Few-Shot Learners
- Time-LLM: Time Series Forecasting by Reprogramming Large Language Models cites Large Language Models Are Zero-Shot Time Series Forecasters
Cited by top-tier papers2
- Transformers as Unsupervised Learning Algorithms: A study on Gaussian MixturesZhiheng Chen, Ruofan Wu, Guanhua FangICLR 2026 · 2 citations
- Pre-trained Large Language Models Learn to Predict Hidden Markov Models In-contextYijia Dai, Zhaolin Gao, Yahya Sattar, Sarah Dean et al.NeurIPS 2025 · 2 citations
Builds on12
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- An Explanation of In-context Learning as Implicit Bayesian InferenceSang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu MaICLR 2022 · 1,030 citations
- Time-LLM: Time Series Forecasting by Reprogramming Large Language ModelsMing Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu et al.ICLR 2024 · 915 citations
- Large Language Models Are Zero-Shot Time Series ForecastersNate Gruver, Marc Finzi, Shikai Qiu, Andrew Gordon WilsonNeurIPS 2023 · 898 citations
Related papers
- What Do Language Models Learn in Context? The Structured Task HypothesisJiaoda Li, Yifan Hou, Mrinmaya Sachan, Ryan CotterellACL 2024 · 5 citations
- Task Descriptors Help Transformers Learn Linear Models In-ContextRuomin Huang, Rong GeICLR 2025
- Language Models Are Implicitly ContinuousSamuele Marro, Davide Evangelista, Xuanqiang Angelo Huang, Emanuele La Malfa et al.ICLR 2025
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder PerspectiveSeungwook Han, Jinyeop Song, Jeff Gore, Pulkit AgrawalICML 2025
- Controllable Context Sensitivity and the Knob Behind ItJulian Minder, Kevin Du, Niklas Stoehr, Giovanni Monea et al.ICLR 2025
