Lune

ACL Findings2025

How do Transformer Embeddings Represent Compositions? A Functional Analysis

Aishik Nagar, Ishaan Singh Rawal, Mansi Dhanania, Cheston Tan

Abstract

Compositionality is a key aspect of human intelligence, essential for reasoning and generalization. While transformer-based models have become the de facto standard for many language modeling tasks, little is known about how they represent compound words, and whether these representations are compositional. In this study, we test compositionality in Mistral, Ope-nAI Large, and Google embedding models, and compare them with BERT. First, we evaluate compositionality in the representations by examining six diverse models of compositionality (addition, multiplication, dilation, regression, etc.). We find that ridge regression, albeit linear, best accounts for compositionality. Surprisingly, we find that the classic vector addition model performs almost as well as any other model. Next, we verify that most embedding models are highly compositional, while BERT shows much poorer compositionality. We verify and visualize our findings with a synthetic dataset consisting of fully transparent adjectivenoun compositions. Overall, we present a thorough investigation of compositionality.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  2. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  3. The Linear Representation Hypothesis and the Geometry of Large Language ModelsKiho Park, Yo Joong Choe, Victor VeitchICML 2024 · 461 citations
  4. Exploring the Role of BERT Token Representations to Explain Sentence Probing ResultsHosein Mohebbi, Ali Modarressi, Mohammad Taher PilehvarEMNLP 2021 · 14 citations
  5. CREPE: Open-Domain Question Answering with False PresuppositionsXinyan Yu, Sewon Min, Luke Zettlemoyer, Hannaneh HajishirziACL 2023 · 13 citations

Related papers

  1. Quantifying Compositionality of Classic and State-of-the-Art EmbeddingsZhijin Guo, Chenhao Xue, Zhaozhen Xu, Hongbo Bo et al.EMNLP Findings 2025
  2. The Paradox of the Compositionality of Natural Language: A Neural Machine Translation Case StudyVerna Dankers, Elia Bruni, Dieuwke HupkesACL 2022
  3. Linear Spaces of Meanings: Compositional Structures in Vision-Language ModelsMatthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille et al.ICCV 2023 · 51 citations
  4. Towards Equipping Transformer with the Ability of Systematic CompositionalityChen Huang, Peixin Qin, Wenqiang Lei, Jiancheng LvAAAI 2024 · 3 citations
  5. Are representations built from the ground up? An empirical examination of local composition in language modelsEmmy Liu, Graham NeubigEMNLP 2022 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines