Lune

ICLR2025Top-tier venue

Specialized Foundation Models Struggle to Beat Supervised Baselines

Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, Mikhail Khodak

Abstract

Following its success for vision and text, the "foundation model" (FM) paradigm-pretraining large models on massive data, then fine-tuning on target tasks-has rapidly expanded to domains in the sciences, engineering, healthcare, and beyond. Has this achieved what the original FMs accomplished, i.e. the supplanting of traditional supervised learning in their domains? To answer we look at three modalities-genomics, satellite imaging, and time series-with multiple recent FMs and compare them to a standard supervised learning workflow: model development, hyperparameter tuning, and training, all using only data from the target task. Across these three specialized domains, we find that it is consistently possible to train simple supervised models-no more complicated than a lightly modified wide ResNet or UNet-that match or even outperform the latest foundation models. Our work demonstrates that the benefits of large-scale pretraining have yet to be realized in many specialized areas, reinforces the need to compare new FMs to strong, well-tuned baselines, and introduces two new, easy-to-use, open-source, and automated workflows for doing so.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • A ConvNet for the 2020s cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • A ConvNet for the 2020s cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Are Transformers Effective for Time Series Forecasting? cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • This Time is Different: An Observability Perspective on Time Series Foundation Models cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • This Time is Different: An Observability Perspective on Time Series Foundation Models cites One Fits All: Power General Time Series Analysis by Pretrained LM
  • One Fits All: Power General Time Series Analysis by Pretrained LM cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • One Fits All: Power General Time Series Analysis by Pretrained LM cites Are Transformers Effective for Time Series Forecasting?
  • CAT: Content-Adaptive Image Tokenization cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. This Time is Different: An Observability Perspective on Time Series Foundation ModelsBen Cohen, Emaad Khwaja, Youssef Doubli, Salahidine Lemaachi et al.NeurIPS 2025 · 68 citations
  2. The Last JITAI? Exploring Large Language Models for Issuing Just-in-Time Adaptive Interventions: Fostering Physical Activity in a Prospective Cardiac Rehabilitation SettingDavid Haag, Devender Kumar, Sebastian Gruber, Dominik P. Hofer et al.CHI 2025 · 35 citations
  3. Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time InteractionJunhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou et al.NeurIPS 2025 · 34 citations
  4. CAT: Content-Adaptive Image TokenizationJunhong Shen, Kushal Tirumala, Michihiro Yasunaga, Ishan Misra et al.NeurIPS 2025 · 17 citations
  5. Parameter-Efficient Adaptation of Geospatial Foundation Models Through Embedding DeflectionRomain Thoreau, Valerio Marsocci, Dawa DerksenICCV 2025 · 1 citation

Builds on30

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. A ConvNet for the 2020sZhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer et al.CVPR 2022 · 6,782 citations
  4. Are Transformers Effective for Time Series Forecasting?Ailing Zeng, Muxi Chen, Lei Zhang, Qiang XuAAAI 2023 · 3,619 citations
  5. One Fits All: Power General Time Series Analysis by Pretrained LMTian Zhou, Peisong Niu, Xue Wang, Liang Sun et al.NeurIPS 2023 · 1,178 citations

Related papers

  1. Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific ModelsRaviteja Vemulapalli, Hadi Pouransari, Fartash Faghri, Sachin Mehta et al.ICML 2024 · 15 citations
  2. Modaltune: Fine-Tuning Slide-Level Foundation Models with Multi-Modal Information for Multi-Task Learning in Digital PathologyVishwesh Ramanathan, Tony Xu, Pushpak Pati, Faruk Ahmed et al.ICCV 2025 · 4 citations
  3. Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-SupervisionYunhe Gao, Yabin Zhang, Chong Wang, Jiaming Liu et al.CVPR 2026
  4. Time-to-Event Pretraining for 3D Medical ImagingZepeng Frazier Huo, Jason Alan Fries, Alejandro Lozano, Jeya Maria Jose Valanarasu et al.ICLR 2025
  5. Towards Foundation Models for Scientific Machine Learning: Characterizing Scaling and Transfer BehaviorShashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji et al.NeurIPS 2023 · 173 citations
Dusk over the sea between two cliffs drawn in fine vertical lines