Semi-Supervised Spoken Language Glossification

Spoken language glossification (SLG) aims to translate the spoken language text into the sign language gloss, i.e., a written record of sign language. In this work, we present a framework named Semi-Supervised Spoken Language Glossification (S 3 LG) for SLG. To tackle the bottleneck of limited parallel data in SLG, our S 3 LG incorporates large-scale monolingual spoken language text into SLG training. The proposed framework follows the selftraining structure that iteratively annotates and learns from pseudo labels. Considering the lexical similarity and syntactic difference between sign language and spoken language, our S 3 LG adopts both the rule-based heuristic and modelbased approach for auto-annotation. During training, we randomly mix these complementary synthetic datasets and mark their differences with a special token. As the synthetic data may be less quality, the S 3 LG further leverages consistency regularization to reduce the negative impact of noise in the synthetic data. Extensive experiments are conducted on public benchmarks to demonstrate the effectiveness of the S 3 LG. Our code is available at https://github.com/yaohj11/S3LG .