Lune

EMNLP Findings2025

FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction

Akriti Jain, Saransh Sharma, Koyel Mukherjee, Soumyabrata Pal

Abstract

Auto-regressive Large Language Models (LLMs) demonstrate remarkable performance across different domains such as vision and language tasks. However, due to sequential processing through multiple transformer layers, autoregressive decoding faces significant computational challenges, particularly in resourceconstrained environments like mobile and edge devices. Existing approaches in literature that aim to improve latency via skipping layers have two distinct flavors: (1) early exit, and (2) input-agnostic heuristics where tokens exit at pre-determined layers irrespective of input sequence. Both the above strategies have limitations, the former cannot be applied in the presence of KV caching, which is essential for speed-ups in modern inference frameworks, and the latter fails to capture variation in layer importance across tasks or, more generally, across input sequences. To address these limitations, we propose FIRST, a model-agnostic framework that reduces inference latency by using layer-specific routers to adaptively skip transformer layers during decoding, based on routing decisions made from the input prompt in the prefill stage. FIRST remains fully compatible with KV caching, enabling faster decoding while maintaining quality. Our method reveals that input adaptivity is essential: Different tasks rely on different subsets of layers to evolve meaningful representations. Extensive experiments show that FIRST significantly reduces latency while outperforming existing layer selection strategies in quality. It retains performance comparable to the base model without skipping. FIRST is thus a promising and efficient solution for LLM deployment in low-resource environments.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BERT Loses Patience: Fast and Robust Inference with Early Exit cites Reducing Transformer Depth on Demand with Structured Dropout
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites Reducing Transformer Depth on Demand with Structured Dropout
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on30

  1. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  2. Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
  3. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations
  4. Reducing Transformer Depth on Demand with Structured DropoutAngela Fan, Edouard Grave, Armand JoulinICLR 2020 · 695 citations
  5. BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley et al.NeurIPS 2020 · 473 citations

Related papers

  1. AdaDecode: Accelerating LLM Decoding with Adaptive Layer ParallelismZhepei Wei, Wei-Lin Chen, Xinyu Zhu, Yu MengICML 2025
  2. River-LLM: Large Language Model Seamless Exit Based on KV ShareYingtao Shen, An ZouACL 2026
  3. EdgeSpec: Distributed Speculative Decoding for Large Language Models at EdgeYulin Chen, Meng Tian, Chao Qiu, Xiaofei Wang et al.INFOCOM 2026
  4. QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV CacheRishabh Tiwari, Haocheng Xi, Aditya Tomar, Coleman Richard Charles Hooper et al.ICML 2025
  5. FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding AccelerationDongwon Jo, Jiwon Song, Yulhwa Kim, Jae-Joon KimACL Findings 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines