BlockDecoder: Boosting ASR Decoders with Context and Merger Modules

Attention-based encoder decoder models remain a popular choice for state-of-the-art automatic speech recognition (ASR). These models combine a powerful audio encoder that extracts rich acoustic features with a decoder that autoregressively produces the ASR output. The decoder handles two critical tasks: (1) building rich text-only context and (2) merging acoustic information from the encoder to ensure the predictions remain faithful to the audio. We observe a systematic pattern across the attention distributions of decoder layers in prior architectures: the initial layers direct most attention towards building textual context, while the later layers largely focus on merging acoustic and textual information for the final predictions. Leveraging this key insight, we propose B LOCK D ECODER , a novel decoder architecture comprising two distinct components: a text encoder that is purely text-based, and a M ERGER that combines information from the audio encoder and text encoder to generate output tokens. Unlike traditional decoders, the M ERGER autoregressively predicts a sequence of K tokens within a block of size K , while relying on the same precomputed contextual information from both text and audio encoders across the block. This design choice allows for the efficient reuse of encoder representations. The separation of the decoder into the text encoder and the M ERGER promotes modularity and more flexible control of parameters via the number of text encoder and M ERGER layers. As a result, B LOCK D ECODER yields a significant speedup ( ∼ 2 x) compared to traditional decoders, across diverse datasets, languages, and speech tasks, without any degradation in performance. The code is available at https://github.com/csalt-research/blockdecoder .