Lune

ACL Findings2023

Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings

Taichi Aida, Danushka Bollegala

Abstract

Languages are dynamic entities, where the meanings associated with words constantly change with time. Detecting the semantic variation of words is an important task for various NLP applications that must make time-sensitive predictions. Existing work on semantic variation prediction have predominantly focused on comparing some form of an averaged contextualised representation of a target word computed from a given corpus. However, some of the previously associated meanings of a target word can become obsolete over time (e.g. meaning of gay as happy), while novel usages of existing words are observed (e.g. meaning of cell as a mobile phone). We argue that mean representations alone cannot accurately capture such semantic variations and propose a method that uses the entire cohort of the contextualised embeddings of the target word, which we refer to as the sibling distribution. Experimental results on SemEval-2020 Task 1 benchmark dataset for semantic variation prediction show that our method outperforms prior work that consider only the mean embeddings, and is comparable to the current state-of-the-art. Moreover, a qualitative analysis shows that our method detects important semantic changes in words that are not captured by the existing methods. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport cites Analysing Lexical Semantic Change with Contextualised Word Representations
  • Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport cites Variance Matters: Detecting Semantic Differences without Corpus/Word Alignment
  • Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language Models cites Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation
  • Variance Matters: Detecting Semantic Differences without Corpus/Word Alignment cites Analysing Lexical Semantic Change with Contextualised Word Representations
  • Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change cites Mind the Gap: Assessing Temporal Generalization in Neural Language Models
  • Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change cites Analysing Lexical Semantic Change with Contextualised Word Representations
  • A New Formulation of Zipf's Meaning-Frequency Law through Contextual Diversity cites Variance Matters: Detecting Semantic Differences without Corpus/Word Alignment
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Evaluating Short-Term Temporal Fluctuations of Social Biases in Social Media Data and Masked Language ModelsYi Zhou, Danushka Bollegala, José Camacho-ColladosEMNLP 2024 · 3 citations
  2. Variance Matters: Detecting Semantic Differences without Corpus/Word AlignmentRyo Nagata, Hiroya Takamura, Naoki Otani, Yoshifumi KawasakiEMNLP 2023 · 2 citations
  3. Learning Dynamic Contextualised Word Embeddings via Template-based Temporal AdaptationXiaohang Tang, Yi Zhou, Danushka BollegalaACL 2023 · 2 citations
  4. Quantifying Lexical Semantic Shift via Unbalanced Optimal TransportRyo Kishino, Hiroaki Yamagiwa, Ryo Nagata, Sho Yokoi et al.ACL 2025
  5. A New Formulation of Zipf's Meaning-Frequency Law through Contextual DiversityRyo Nagata, Kumiko Tanaka-IshiiACL 2025

Builds on4

  1. Mind the Gap: Assessing Temporal Generalization in Neural Language ModelsAngeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal et al.NeurIPS 2021 · 315 citations
  2. Analysing Lexical Semantic Change with Contextualised Word RepresentationsMario Giulianelli, Marco Del Tredici, Raquel FernándezACL 2020 · 118 citations
  3. Information-Theoretic Probing for Linguistic StructureTiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod et al.ACL 2020 · 21 citations
  4. Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic ChangeZhaochen Su, Zecheng Tang, Xinyan Guan, Lijun Wu et al.EMNLP 2022 · 11 citations

Related papers

  1. A Word Sense Distribution-based approach for Semantic Change PredictionXiaohang Tang, Yi Zhou, Taichi Aida, Procheta Sen et al.EMNLP Findings 2023 · 2 citations
  2. Swap and Predict - Predicting the Semantic Changes in Words across Corpora by Context SwappingTaichi Aida, Danushka BollegalaEMNLP Findings 2023 · 2 citations
  3. Sequential Modelling of the Evolution of Word Representations for Semantic Change DetectionAdam Tsakalidis, Maria LiakataEMNLP 2020 · 14 citations
  4. A Semantic Distance Metric Learning approach for Lexical Semantic Change DetectionTaichi Aida, Danushka BollegalaACL Findings 2024 · 4 citations
  5. Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding TasksMaurício Gruppi, Pin-Yu Chen, Sibel AdaliAAAI 2021 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines