ACL Findings2024
Unsupervised Sign Language Translation and Generation
Zhengsheng Guo, Zhiwei He, Wenxiang Jiao, Xing Wang, Rui Wang, Kehai Chen, Zhaopeng Tu, Yong Xu, Min Zhang
Abstract
Motivated by the success of unsupervised neural machine translation (UNMT), we introduce an unsupervised sign language translation and generation network (USLNet), which learns from abundant single-modality (text and video) data without parallel sign language data. USLNet comprises two main components: single-modality reconstruction modules (text and video) that rebuild the input from its noisy version in the same modality and cross-modality back-translation modules (text-video-text and video-text-video) that reconstruct the input from its noisy version in the different modality using back-translation procedure. Unlike the single-modality backtranslation procedure in text-based UNMT, USLNet faces the cross-modality discrepancy in feature representation, in which the length and the feature dimension mismatch between text and video sequences. We propose a sliding window method to address the issues of aligning variable-length text with video sequences. To our knowledge, USLNet is the first unsupervised sign language translation and generation model capable of generating both natural language text and sign language video in a unified manner. Experimental results on the BBC-Oxford Sign Language dataset and Open-Domain American Sign Language dataset reveal that USLNet achieves competitive results compared to supervised baseline models, indicating its effectiveness in sign language translation and generation. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation cites Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation
- Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation cites Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation cites Learning Transferable Visual Models From Natural Language Supervision
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation cites Cross-modality Data Augmentation for End-to-End Sign Language Translation
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation cites Open-Domain Sign Language Translation Learned from Online Video
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation cites A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation
- Cross-modality Data Augmentation for End-to-End Sign Language Translation cites A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation
- Open-Domain Sign Language Translation Learned from Online Video cites A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation
- A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers2
- Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language TranslationEdward Fish, Richard BowdenNeurIPS 2025 · 15 citations
- Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language TranslationYasser Hamidullah, Koel Dutta Chowdhury, Yusser Al Ghussin, Shakib Yazdani et al.ICLR 2026 · 3 citations
Builds on10
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- A Simple Multi-Modality Transfer Learning Baseline for Sign Language TranslationYutong Chen, Fangyun Wei, Xiao Sun, Zhirong Wu et al.CVPR 2022 · 137 citations
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language TranslationRyan Wong, Necati Cihan Camgöz, Richard BowdenICLR 2024 · 58 citations
- Open-Domain Sign Language Translation Learned from Online VideoBowen Shi, Diane Brentari, Gregory Shakhnarovich, Karen LivescuEMNLP 2022 · 39 citations
- Cross-modality Data Augmentation for End-to-End Sign Language TranslationJinhui Ye, Wenxiang Jiao, Xing Wang, Zhaopeng Tu et al.EMNLP Findings 2023 · 26 citations
Related papers
- SLTUNET: A Simple Unified Model for Sign Language TranslationBiao Zhang, Mathias Müller, Rico SennrichICLR 2023 · 14 citations
- Listen, Decipher and Sign: Toward Unsupervised Speech-to-Sign Language RecognitionLiming Wang, Junrui Ni, Heting Gao, Jialu Li et al.ACL Findings 2023 · 1 citation
- Leveraging the Power of MLLMs for Gloss-Free Sign Language TranslationJungeun Kim, Hyeongwoo Jeon, Jongseong Bae, Ha Young KimICCV 2025 · 10 citations
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal AlignmentRui Zhao, Liang Zhang, Biao Fu, Cong Hu et al.AAAI 2024 · 36 citations
- LLMs are Good Sign Language TranslatorsJia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani et al.CVPR 2024
