Stable Signer: Hierarchical Sign Language Generative Model

Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages 1 , and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multistyle sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods, which is a significant increase in the SLP field. * Equal Contribution. 1 Gloss represents the specific gesture text words. The students will study in the library tomorrow. DWPose Denoising Off STUDENT LIBRARY STUDY TOMORROW Our clean pose input can be trained end-to-end with high quality. Random poses can cause noise that leads to blurriness. Introducing rule priors can reduce posture flickering. Minimizing the instability of the posture can reduce the burden of optimization. Denoising ON The semantic understanding usually involves Prompt2Gloss, Text2Gloss, and Gloss is generally in capitalized text.