Lune

ACL Findings2023

Bridging the Granularity Gap for Acoustic Modeling

Chen Xu, Yuhao Zhang, Chengbo Jiao, Xiaoqian Liu, Chi Hu, Xin Zeng, Tong Xiao, Anxiang Ma, Huizhen Wang, Jingbo Zhu

Abstract

While Transformer has become the de-facto standard for speech, modeling upon the finegrained frame-level features remains an open challenge of capturing long-distance dependencies and distributing the attention weights. We propose Progressive Down-Sampling (PDS) which gradually compresses the acoustic features into coarser-grained units containing more complete semantic information, like textlevel representation. In addition, we develop a representation fusion method to alleviate information loss that occurs inevitably during high compression. In this way, we compress the acoustic features into 1/32 of the initial length while achieving better or comparable performances on the speech recognition task. And as a bonus, it yields inference speedups ranging from 1.20× to 1.47×. By reducing the modeling burden, we also achieve competitive results when training on the more challenging speech translation task 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CTC-based Non-autoregressive Speech Translation cites Squeezeformer: An Efficient Transformer for Automatic Speech Recognition
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. CTC-based Non-autoregressive Speech TranslationChen Xu, Xiaoqian Liu, Xiaowen Liu, Qingxuan Sun et al.ACL 2023 · 4 citations

Builds on8

  1. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without ConvolutionsWenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan et al.ICCV 2021 · 4,909 citations
  2. Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language ProcessingZihang Dai, Guokun Lai, Yiming Yang, Quoc LeNeurIPS 2020 · 273 citations
  3. Squeezeformer: An Efficient Transformer for Automatic Speech RecognitionSehoon Kim, Amir Gholami, Albert E. Shaw, Nicholas Lee et al.NeurIPS 2022 · 152 citations
  4. Consecutive Decoding for Speech-to-text TranslationQianqian Dong, Mingxuan Wang, Hao Zhou, Shuang Xu et al.AAAI 2021 · 46 citations
  5. Adaptive Feature Selection for End-to-End Speech TranslationBiao Zhang, Ivan Titov, Barry Haddow, Rico SennrichEMNLP Findings 2020 · 33 citations

Related papers

  1. Explicit Sentence Compression for Neural Machine TranslationZuchao Li, Rui Wang, Kehai Chen, Masao Utiyama et al.AAAI 2020 · 31 citations
  2. DiTReducio: A Training-Free Acceleration for DiT-Based TTS via Progressive CalibrationYanru Huo, Ziyue Jiang, Zuoli Tang, Qingyang Hong et al.ACL Findings 2026
  3. Speech Token Prediction via Compressed-to-fine Language Modeling for Speech GenerationWenrui Liu, Qian Chen, Wen Wang, Guanrou Yang et al.ACM MM 2025
  4. Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language ModelsBajian Xiang, Tingwei Guo, Xuan Chen, Yang HanACL Findings 2026
  5. DiffS2UT: A Semantic Preserving Diffusion Model for Textless Direct Speech-to-Speech TranslationYongxin Zhu, Zhujin Gao, Xinyuan Zhou, Zhongyi Ye et al.EMNLP 2023 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines