Lune

ACM MM2020Top-tier venue

Adaptive Temporal Triplet-loss for Cross-modal Embedding Learning

David Semedo, João Magalhães

Abstract

There are many domains where the temporal dimension is critical to unveil how different modalities, such as images and texts, are correlated. Notably, in the social media domain, information is constantly evolving over time according to the events that take place in the real world. In this work, we seek for highly expressive loss functions that allow the encoding of data temporal traits into cross-modal embedding spaces. To achieve this goal, we propose to steer the learning procedure of such embedding through a set of adaptively enforced temporal constraints. In particular, we propose a new formulation of the triplet loss function, where the traditional static margin is superseded by a novel temporally adaptive maximum margin function. This novel redesign of the static margin formulation, allows the embedding to effectively capture not only the semantic correlations across data modalities, but also data's fine-grained temporal correlations. Our experiments confirm the effectiveness of our model in structuring different modalities, while organizing data according to temporal correlations. Moreover, we experimentally highlight how can these embeddings be used for multimedia understanding.

Ask Lune

Know where this paper stands in minutes, not an afternoon.

Lune can read the top-tier papers that cite it and the ones closest to it, and tells you what it claims, what came after and where it falls short, each point cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

its abstract · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Assisting News Media Editors with Cohesive Visual StorylinesGonçalo Marcelino, David Semedo, André Mourão, Saverio G. Blasi et al.ACM MM 2021

Related papers

  1. ASMR: Learning Attribute-Based Person Search with Adaptive Semantic Margin RegularizerBoseung Jeong, Jicheol Park, Suha KwakICCV 2021 · 29 citations
  2. Exploring Graph-Structured Semantics for Cross-Modal RetrievalLei Zhang, Leiting Chen, Chuan Zhou, Fan Yang et al.ACM MM 2021 · 14 citations
  3. Learning Semantic Relationship among Instances for Image-Text MatchingZheren Fu, Zhendong Mao, Yan Song, Yongdong ZhangCVPR 2023
  4. Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment AnalysisYing Zeng, Sijie Mai, Haifeng HuEMNLP Findings 2021 · 27 citations
  5. Time-aware Entity Alignment using Temporal Relational AttentionChengjin Xu, Fenglong Su, Bo Xiong, Jens LehmannWWW 2022 · 47 citations
Dusk over the sea between two cliffs drawn in fine vertical lines