Lune

EMNLP Findings2024

The Shape of Word Embeddings: Quantifying Non-Isometry with Topological Data Analysis

Ondrej Draganov, Steven Skiena

Abstract

Word embeddings represent language vocabularies as clouds of d-dimensional points. We investigate how information is conveyed by the general shape of these clouds, instead of representing the semantic meaning of each token. Specifically, we use the notion of persistent homology from topological data analysis (TDA) to measure the distances between language pairs from the shape of their unlabeled embeddings. These distances quantify the degree of non-isometry of the embeddings. To distinguish whether these differences are random training errors or capture real information about the languages, we use the computed distance matrices to construct language phylogenetic trees over 81 Indo-European languages. Careful evaluation shows that our reconstructed trees exhibit strong and statistically-significant similarities to the reference.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on1

  1. Are All Good Word Vector Spaces Isomorphic?Ivan Vulic, Sebastian Ruder, Anders SøgaardEMNLP 2020 · 7 citations

Related papers

  1. Improving Self-supervised Molecular Representation Learning using Persistent HomologyYuankai Luo, Lei Shi, Veronika ThostNeurIPS 2023 · 13 citations
  2. TOPOGRAPH: Topology-Preserving Graph Reduction with Adaptive Structure for Persistent HomologyZonghao Chen, Yuncheng Jiang, Gang LiAAAI 2026
  3. Representation Topology Divergence: A Method for Comparing Neural Network RepresentationsSerguei Barannikov, Ilya Trofimov, Nikita Balabin, Evgeny BurnaevICML 2022 · 69 citations
  4. Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language RepresentationsArturo Oncevay, Barry Haddow, Alexandra BirchEMNLP 2020 · 1 citation
  5. Mapping the Multiverse of Latent RepresentationsJeremy Wayland, Corinna Coupette, Bastian RieckICML 2024 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines