Lune

ICLR2025Top-tier venue

Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity

Mutian He, Philip N. Garner

Abstract

Architectures such as Linformer and Mamba have recently emerged as competitive linear time replacements for transformers. However, corresponding large pretrained models are often unavailable, especially in non-text domains. To remedy this, we present a Cross-Architecture Layerwise Distillation (CALD) approach that jointly converts a transformer model to a linear time substitute and fine-tunes it to a target task. We also compare several means to guide the fine-tuning to optimally retain the desired inference capability from the original model. The methods differ in their use of the target model and the trajectory of the parameters. In a series of empirical studies on language processing, language modeling, and speech processing, we show that CALD can effectively recover the result of the original model, and that the guiding strategy contributes to the result. Some reasons for the variation are suggested.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention cites Reformer: The Efficient Transformer
  • Big Bird: Transformers for Longer Sequences cites Reformer: The Efficient Transformer
  • Efficiently Modeling Long Sequences with Structured State Spaces cites Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on22

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Efficiently Modeling Long Sequences with Structured State SpacesAlbert Gu, Karan Goel, Christopher RéICLR 2022 · 3,482 citations
  3. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  4. Reformer: The Efficient TransformerNikita Kitaev, Lukasz Kaiser, Anselm LevskayaICLR 2020 · 2,878 citations
  5. Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionAngelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François FleuretICML 2020 · 2,665 citations

Related papers

  1. The Mamba in the Llama: Distilling and Accelerating Hybrid ModelsJunxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush et al.NeurIPS 2024 · 146 citations
  2. ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision ModelsGuoyizhe Wei, Rama ChellappaICCV 2025
  3. MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language ModelingYingyue Li, Bencheng Liao, Wenyu Liu, Xinggang WangICCV 2025 · 1 citation
  4. Effective Distillation to Hybrid xLSTM ArchitecturesLukas Hauzenberger, Niklas Schmidinger, Thomas Schmied, Anamaria-Roberta Hartl et al.ICML 2026 · 3 citations
  5. Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic ModelsAviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter et al.NeurIPS 2024 · 78 citations
Dusk over the sea between two cliffs drawn in fine vertical lines