Lune

EMNLP Findings2023

Swap and Predict - Predicting the Semantic Changes in Words across Corpora by Context Swapping

Taichi Aida, Danushka Bollegala

Abstract

Meanings of words change over time and across domains. Detecting the semantic changes of words is an important task for various NLP applications that must make time-sensitive predictions. We consider the problem of predicting whether a given target word, w, changes its meaning between two different text corpora, C 1 and C 2 . For this purpose, we propose Swappingbased Semantic Change Detection (SSCD), an unsupervised method that randomly swaps contexts between C 1 and C 2 where w occurs. We then look at the distribution of contextualised word embeddings of w, obtained from a pretrained masked language model (MLM), representing the meaning of w in its occurrence contexts in C 1 and C 2 . Intuitively, if the meaning of w does not change between C 1 and C 2 , we would expect the distributions of contextualised word embeddings of w to remain the same before and after this random swapping process. Despite its simplicity, we demonstrate that even by using pretrained MLMs without any fine-tuning, our proposed context swapping method accurately predicts the semantic changes of words in four languages (English, German, Swedish, and Latin) and across different time spans (over 50 years and about five years). Moreover, our method achieves significant performance improvements compared to strong baselines for the English semantic change prediction task. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change cites Mind the Gap: Assessing Temporal Generalization in Neural Language Models
  • Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change cites Analysing Lexical Semantic Change with Contextualised Word Representations
  • Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings cites Mind the Gap: Assessing Temporal Generalization in Neural Language Models
  • Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings cites Analysing Lexical Semantic Change with Contextualised Word Representations
  • Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings cites Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. HistLens: Mapping Idea Change across Concepts and CorporaYi Jing, Weiyun Qiu, Yihang Peng, Zhifang SuiACL 2026

Builds on4

  1. Mind the Gap: Assessing Temporal Generalization in Neural Language ModelsAngeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal et al.NeurIPS 2021 · 315 citations
  2. Analysing Lexical Semantic Change with Contextualised Word RepresentationsMario Giulianelli, Marco Del Tredici, Raquel FernándezACL 2020 · 118 citations
  3. Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic ChangeZhaochen Su, Zecheng Tang, Xinyan Guan, Lijun Wu et al.EMNLP 2022 · 11 citations
  4. Unsupervised Semantic Variation Prediction using the Distribution of Sibling EmbeddingsTaichi Aida, Danushka BollegalaACL Findings 2023 · 4 citations

Related papers

  1. A Word Sense Distribution-based approach for Semantic Change PredictionXiaohang Tang, Yi Zhou, Taichi Aida, Procheta Sen et al.EMNLP Findings 2023 · 2 citations
  2. A Semantic Distance Metric Learning approach for Lexical Semantic Change DetectionTaichi Aida, Danushka BollegalaACL Findings 2024 · 4 citations
  3. Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding TasksMaurício Gruppi, Pin-Yu Chen, Sibel AdaliAAAI 2021 · 7 citations
  4. Sequential Modelling of the Evolution of Word Representations for Semantic Change DetectionAdam Tsakalidis, Maria LiakataEMNLP 2020 · 14 citations
  5. Definition generation for lexical semantic change detectionMariia Fedorova, Andrey Kutuzov, Yves ScherrerACL Findings 2024 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines