ACL Findings2025
Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCR
Chan-Jan Hsu, Yi-Chang Chen, Feng-Ting Liao, Pei-Chen Ho, Yu-Hsiang Wang, Po-Chun Hsu, Da-shan Shiu
Abstract
We propose "Generative Fusion Decoding" (GFD), a novel shallow fusion framework designed to integrate large language models (LLMs) into cross-modal text recognition systems for automatic speech recognition (ASR) and optical character recognition (OCR). We derive the necessary formulations to enable GFD to operate across mismatched token spaces of different models by calculating likelihood at the byte level, thereby enabling seamless fusion and synchronous progression during the decoding process. GFD is plug-and-play by design, making it readily compatible with various auto-regressive models without the need for any re-training. GFD proves effective for general ASR and OCR tasks through intermediate and frequent interactions with LLMs, surpassing cascaded methods in English and Mandarin benchmarks. In addition, GFD transfers in-context learning abilities of LLMs and allows for adaptive ASR in instruction-aware and long-context settings, yielding significant WER reductions of up to 17.7%. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- PaLM-E: An Embodied Multimodal Language Model cites Flamingo: a Visual Language Model for Few-Shot Learning
- Robust Speech Recognition via Large-Scale Weak Supervision cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites PaLM-E: An Embodied Multimodal Language Model
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Builds on10
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch et al.ICML 2023 · 2,601 citations
Related papers
- It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech RecognitionChen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi et al.ICLR 2024 · 37 citations
- Speech Recognition Meets Large Language Model: Benchmarking, Models, and ExplorationZiyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao et al.AAAI 2025 · 17 citations
- LaRA: Large Rank Adaptation for Speech and Text Cross-Modal Learning in Large Language ModelsZuhair Hasan Shaik, Pradyoth Hegde, Prashant Bannulmath, Deepak K. T.EMNLP Findings 2024 · 1 citation
- SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative DecodingLinye Wei, Shuzhang Zhong, Songqiang Xu, Runsheng Wang et al.DAC 2025 · 7 citations
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned RepresentationsJiaming Han, Hao Chen, Yang Zhao, Hanyu Wang et al.NeurIPS 2025 · 50 citations
