CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training

Speech or text representation generated by pre-trained models contains modal-specific information that could be combined for benefiting spoken language understanding (SLU) tasks.In this work, we propose a novel pretraining paradigm termed Continuous Integrateand-Fire Pre-Training (CIF-PT).It relies on a simple but effective frame-to-token alignment: continuous integrate-and-fire (CIF) to bridge the representations between speech and text.It jointly performs speech-to-text training and language model distillation through CIF as the pretraining (PT).Evaluated on SLU benchmark SLURP dataset, CIF-PT outperforms the stateof-the-art model by 1.94% of accuracy and 2.71% of SLU-F1 on the tasks of intent classification and slot filling, respectively.We also observe the cross-modal representation extracted by CIF-PT obtains better performance than other neural interfaces for the tasks of SLU, including the dominant speech representation learned from self-supervised pre-training. * Equal contribution.