Lune

ACL Findings2022

Prior Knowledge and Memory Enriched Transformer for Sign Language Translation

Tao Jin, Zhou Zhao, Meng Zhang, Xingshan Zeng

Abstract

This paper attacks the challenging problem of sign language translation (SLT), which involves not only visual and textual understanding but also additional prior knowledge learning (i.e. performing style, syntax). However, the majority of existing methods with vanilla encoder-decoder structures fail to sufficiently explore all of them. Based on this concern, we propose a novel method called Prior knowledge and memory Enriched Transformer (PET) for SLT, which incorporates the auxiliary information into vanilla transformer. Concretely, we develop gated interactive multi-head attention which associates the multimodal representation and global signing style with adaptive gated functions. One Part-of-Speech (POS) sequence generator relies on the associated information to predict the global syntactic structure, which is thereafter leveraged to guide the sentence generation. Besides, considering that the visualtextual context information, and additional auxiliary knowledge of a word may appear in more than one video, we design a multi-stream memory structure to obtain higher-quality translations, which stores the detailed correspondence between a word and its various relevant information, leading to a more comprehensive understanding for each word. We conduct extensive empirical studies on RWTH-PHOENIX-Weather-2014T dataset with both signer-dependent and signer-independent conditions. The quantitative and qualitative experimental results comprehensively reveal the effectiveness of PET.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Sign Language Translation with Iterative Prototype cites Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
  • Sign Language Translation with Iterative Prototype cites TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
  • SLTUNET: A Simple Unified Model for Sign Language Translation cites Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
  • SLTUNET: A Simple Unified Model for Sign Language Translation cites TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
  • Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation cites Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
  • Exploring Group Video Captioning with Efficient Relational Approximation cites Contrastive Disentangled Meta-Learning for Signer-Independent Sign Language Translation
  • Gloss-Free End-to-End Sign Language Translation cites Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
  • Gloss-Free End-to-End Sign Language Translation cites TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
  • TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation cites Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
  • Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation cites Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation
  • Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation cites SLTUNET: A Simple Unified Model for Sign Language Translation
  • Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation cites Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation
  • Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation cites TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Sign Language Translation with Iterative PrototypeHuijie Yao, Wengang Zhou, Hao Feng, Hezhen Hu et al.ICCV 2023 · 31 citations
  2. Gloss-Free End-to-End Sign Language TranslationKezhou Lin, Xiaohan Wang, Linchao Zhu, Ke Sun et al.ACL 2023 · 25 citations
  3. Exploring Group Video Captioning with Efficient Relational ApproximationWang Lin, Tao Jin, Ye Wang, Wenwen Pan et al.ICCV 2023 · 17 citations
  4. SLTUNET: A Simple Unified Model for Sign Language TranslationBiao Zhang, Mathias Müller, Rico SennrichICLR 2023 · 14 citations
  5. Leveraging the Power of MLLMs for Gloss-Free Sign Language TranslationJungeun Kim, Hyeongwoo Jeon, Jongseong Bae, Ha Young KimICCV 2025 · 10 citations

Builds on4

  1. TSPNet: Hierarchical Feature Learning via Temporal Semantic Pyramid for Sign Language TranslationDongxu Li, Chenchen Xu, Xin Yu, Kaihao Zhang et al.NeurIPS 2020 · 171 citations
  2. Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL TranslationAndreas Voskou, Konstantinos P. Panousis, Dimitrios I. Kosmopoulos, Dimitris N. Metaxas et al.ICCV 2021 · 51 citations
  3. Contrastive Disentangled Meta-Learning for Signer-Independent Sign Language TranslationTao Jin, Zhou ZhaoACM MM 2021 · 23 citations
  4. Sign Language Transformers: Joint End-to-End Sign Language Recognition and TranslationNecati Cihan Camgöz, Oscar Koller, Simon Hadfield, Richard BowdenCVPR 2020

Related papers

  1. Human Part-wise 3D Motion Context Learning for Sign Language RecognitionTaeryung Lee, Yeonguk Oh, Kyoung Mu LeeICCV 2023 · 14 citations
  2. Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal AlignmentRui Zhao, Liang Zhang, Biao Fu, Cong Hu et al.AAAI 2024 · 36 citations
  3. Gloss Attention for Gloss-free Sign Language TranslationAoxiong Yin, Tianyun Zhong, Li Tang, Weike Jin et al.CVPR 2023
  4. Two-Stream Network for Sign Language Recognition and TranslationYutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu et al.NeurIPS 2022 · 288 citations
  5. CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition with Variational AlignmentJiangbin Zheng, Yile Wang, Cheng Tan, Siyuan Li et al.CVPR 2023
Dusk over the sea between two cliffs drawn in fine vertical lines