Lune

ACL Findings2023

Contrastive Token-Wise Meta-Learning for Unseen Performer Visual Temporal-Aligned Translation

Linjun Li, Tao Jin, Xize Cheng, Ye Wang, Wang Lin, Rongjie Huang, Zhou Zhao

Abstract

Visual temporal-aligned translation aims to transform the visual sequence into natural words, including important applicable tasks such as lipreading and fingerspelling recognition. However, various performance habits of specific words by different speakers or signers can lead to visual ambiguity, which has become a major obstacle to the development of current methods. Considering the constraints above, the generalization ability of the translation system is supposed to be further explored through the evaluation results on unseen performers. In this paper, we develop a novel generalizable framework named Contrastive Token-Wise Meta-learning (CtoML), which strives to transfer recognition skills to unseen performers. To the best of our knowledge, employing meta-learning methods directly in the image domain poses two main challenges, and we propose corresponding strategies. First, sequence prediction in visual temporal-aligned translation, which aims to generate multiple words autoregressively, is different from the vanilla classification. Thus, we devise the token-wise diversity-aware weights for the meta-train stage, which encourages the model to make efforts on those ambiguously recognized tokens. Second, considering the consistency of word-visual prototypes across different domains, we develop two complementary global and local contrastive losses to maintain inter-class relationships and promote domainindependence. We conduct extensive experiments on the widely-used lipreading dataset GRID and the fingerspelling dataset ChicagoF-SWild, and the experimental results show the effectiveness of our proposed CtoML over existing state-of-the-art methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Uni-Dubbing: Zero-Shot Speech Synthesis from Visual ArticulationSongju Lei, Xize Cheng, Mengjiao Lyu, Jianqiao Hu et al.ACL 2024 · 1 citation
  2. PACHAT: Persona-Aware Speech Assistant for Multi-party DialogueDongjie Fu, Xize Cheng, Linjun Li, Xiaoda Yang et al.EMNLP 2025
  3. OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned SynthesisJunuk Cha, Jihyeon Kim, Han-Mu ParkCVPR 2026
  4. BANZ-FS: BANZSL Fingerspelling DatasetXin Shen, Yan Ke, Xinyu Wang, Xin YuICLR 2026

Builds on18

  1. Domain Generalization with MixStyleKaiyang Zhou, Yongxin Yang, Yu Qiao, Tao XiangICLR 2021 · 986 citations
  2. Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion ModelsRongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren et al.ICML 2023 · 469 citations
  3. Learning Audio-Visual Speech Representation by Masked Multimodal Cluster PredictionBowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman MohamedICLR 2022 · 460 citations
  4. Domain Generalization using Causal MatchingDivyat Mahajan, Shruti Tople, Amit SharmaICML 2021 · 399 citations
  5. Hearing Lips: Improving Lip Reading by Distilling Speech RecognizersYa Zhao, Rui Xu, Xinchao Wang, Peng Hou et al.AAAI 2020 · 106 citations

Related papers

  1. Contrastive Disentangled Meta-Learning for Signer-Independent Sign Language TranslationTao Jin, Zhou ZhaoACM MM 2021 · 23 citations
  2. Seeing What You Said: Talking Face Generation Guided by a Lip Reading ExpertJiadong Wang, Xinyuan Qian, Malu Zhang, Robby T. Tan et al.CVPR 2023
  3. SyncTalklip: Highly Synchronized Lip-Readable Speaker Generation with Multi-Task LearningXiaoda Yang, Xize Cheng, Dongjie Fu, Minghui Fang et al.ACM MM 2024 · 4 citations
  4. MC-SLT: Towards Low-Resource Signer-Adaptive Sign Language TranslationTao Jin, Zhou Zhao, Meng Zhang, Xingshan ZengACM MM 2022 · 14 citations
  5. CALLip: Lipreading using Contrastive and Attribute LearningYiyang Huang, Xuefeng Liang, Chaowei FangACM MM 2021 · 14 citations
Dusk over the sea between two cliffs drawn in fine vertical lines