Adaptive Computation with Elastic Input Sequence
Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, Yang You
Abstract
Humans have the ability to adapt the type of information they use, the procedure they employ, and the amount of time they spend when solving problems. However, most standard neural networks have a fixed function type and computation budget regardless of the sample's nature or difficulty. Adaptivity is a powerful paradigm as it not only imbues practitioners with flexibility pertaining to the downstream usage of these models but can also serve as a powerful inductive bias for solving certain challenging classes of problems. In this work, we introduce a new approach called AdaTape, which allows for dynamic computation in neural networks through adaptive tape tokens. AdaTape utilizes an elastic input sequence by equipping an architecture with a dynamic read-and-write tape. Specifically, we adaptively generate input sequences using tape tokens obtained from a tape bank which can be either trainable or derived from input data. We examine the challenges and requirements to obtain dynamic sequence content and length, and propose the Adaptive Tape Reading (ATR) algorithm to achieve both goals. Through extensive experiments on image recognition tasks, we show that AdaTape can achieve better performance while maintaining the computational cost. To facilitate further research, we have released code at https://github.com/google-research/scenic .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
9 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens cites Vision Transformers Need Registers
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens cites Scaling Vision with Sparse Mixture of Experts
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens cites Scaling Vision Transformers to 22 Billion Parameters
- Think before you speak: Training Language Models With Pause Tokens cites Vision Transformers Need Registers
- Vision Transformers Need Registers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Scaling Vision with Sparse Mixture of Experts cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Scaling Vision with Sparse Mixture of Experts cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
- Scaling Vision with Sparse Mixture of Experts cites Training data-efficient image transformers & distillation through attention
- The Pitfalls of Next-Token Prediction cites Think before you speak: Training Language Models With Pause Tokens
- Scaling Vision Transformers to 22 Billion Parameters cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Scaling Vision Transformers to 22 Billion Parameters cites Scaling Vision with Sparse Mixture of Experts
- TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy cites Think before you speak: Training Language Models With Pause Tokens
- TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy cites Vision Transformers Need Registers
- Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Cited by top-tier papers9
- Vision Transformers Need RegistersTimothée Darcet, Maxime Oquab, Julien Mairal, Piotr BojanowskiICLR 2024 · 769 citations
- Think before you speak: Training Language Models With Pause TokensSachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon et al.ICLR 2024 · 240 citations
- The Pitfalls of Next-Token PredictionGregor Bachmann, Vaishnavh NagarajanICML 2024 · 163 citations
- TabPedia: Towards Comprehensive Visual Table Understanding with Concept SynergyWeichao Zhao, Hao Feng, Qi Liu, Jingqun Tang et al.NeurIPS 2024 · 97 citations
- Mixture of Nested Experts: Adaptive Processing of Visual TokensGagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani et al.NeurIPS 2024 · 29 citations
Builds on17
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
- RandAugment: Practical Automated Data Augmentation with a Reduced Search SpaceEkin Dogus Cubuk, Barret Zoph, Jonathon Shlens, Quoc LeNeurIPS 2020 · 4,453 citations
- Scaling Vision with Sparse Mixture of ExpertsCarlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann et al.NeurIPS 2021 · 1,213 citations
- Scaling Vision Transformers to 22 Billion ParametersMostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski et al.ICML 2023 · 848 citations
Related papers
- Rethinking Addressing in Language Models via Contextualized Equivariant Positional EncodingJiajun Zhu, Peihao Wang, Ruisi Cai, Jason D. Lee et al.ICML 2025
- Adaptive Length Image Tokenization via Recurrent AllocationShivam Duggal, Phillip Isola, Antonio Torralba, William T. FreemanICLR 2025 · 1 citation
- DAPE: Data-Adaptive Positional Encoding for Length ExtrapolationChuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang et al.NeurIPS 2024 · 42 citations
- Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image RecognitionYulin Wang, Rui Huang, Shiji Song, Zeyi Huang et al.NeurIPS 2021 · 283 citations
- ElasticTok: Adaptive Tokenization for Image and VideoWilson Yan, Volodymyr Mnih, Aleksandra Faust, Matei Zaharia et al.ICLR 2025
