Towards Speech Dialogue Translation Mediating Speakers of Different Languages

We present a new task, speech dialogue translation mediating speakers of different languages. We construct the SpeechBSD dataset for the task and conduct baseline experiments. Furthermore, we consider context to be an important aspect that needs to be addressed in this task and propose two ways of utilizing context, namely monolingual context and bilingual context. We conduct cascaded speech translation experiments using Whisper and mBART, and show that bilingual context performs better in our settings. 彼は良い考えだと言ってました He said it's a good idea MT 少し甘いと思います I think it's a bit sweet MT What do you think about it? あなたはどう思いますか MT ASR 𝑈 ! 𝑈 " 𝑈 # 𝑋 ! 𝑌 ! ! 𝑌 ! " 𝑌 # " 𝑌 " ! Japanese Speaker 𝐿 ! 𝑆 ! English Speaker 𝐿 " 𝑆 " Should be translated as "naive" ASR 𝑋 # 𝑌 # !