EMNLP Findings2023
Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models
Arya McCarthy, Hao Zhang, Shankar Kumar, Felix Stahlberg, Ke Wu
Abstract
One challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. To address this mismatch, we adapt large language models (LLMs) to split long ASR transcripts into segments that can be independently translated so as to maximize the overall translation quality. We overcome the tendency of hallucination in LLMs by incorporating finite-state constraints during decoding; these eliminate invalid outputs without requiring additional training. We discover that LLMs are adaptable to transcripts containing ASR errors through prompt-tuning or fine-tuning. Relative to a state-of-the-art automatic punctuation baseline, our best LLM improves the average BLEU by 2.9 points for English-German, English-Spanish, and English-Arabic TED talk translation in 9 test sets, just by improving segmentation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on5
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations
- Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented SpeechKatrin Tomanek, Vicky Zayats, Dirk Padfield, Kara Vaillancourt et al.EMNLP 2021 · 37 citations
- Better Chinese Sentence Segmentation with Reinforcement LearningSrivatsan Srinivasan, Chris DyerACL Findings 2021 · 4 citations
- A unified approach to sentence segmentation of punctuated text in many languagesRachel Wicks, Matt PostACL 2021
Related papers
- SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech TranslationKeqi Deng, Wenxi Chen, Xie Chen, Philip C. WoodlandACL 2025
- From Tens of Hours to Tens of Thousands: Scaling Back-Translation for Speech RecognitionTianduo Wang, Lu Xu, Wei Lu, Shanbo ChengEMNLP 2025 · 1 citation
- Learning Adaptive Segmentation Policy for Simultaneous TranslationRuiqing Zhang, Chuanqiang Zhang, Zhongjun He, Hua Wu et al.EMNLP 2020 · 41 citations
- Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language ModelsYuchen Hu, Chen Chen, Chengwei Qin, Qiushi Zhu et al.ACL Findings 2024 · 11 citations
- Contrastive Learning for Task-Independent SpeechLLM-PretrainingMaike Züfle, Jan NiehuesACL Findings 2025
