Lune

ACL Findings2025

On Entity Identification in Language Models

Masaki Sakata, Benjamin Heinzerling, Sho Yokoi, Takumi Ito, Kentaro Inui

Abstract

We analyze the extent to which internal representations of language models (LMs) identify and distinguish mentions of named entities, focusing on the many-to-many correspondence between entities and their mentions. We first formulate two problems of entity mentions -ambiguity and variabilityand propose a framework analogous to clustering quality metrics. Specifically, we quantify through cluster analysis of LM internal representations the extent to which mentions of the same entity cluster together and mentions of different entities remain separated. Our experiments examine five Transformer-based autoregressive models, showing that they effectively identify and distinguish entities with metrics analogous to precision and recall ranging from 0.66 to 0.9. Further analysis reveals that entity-related information is compactly represented in a low-dimensional linear subspace at early LM layers. Additionally, we clarify how the characteristics of entity representations influence word prediction performance. These findings are interpreted through the lens of isomorphism between LM representations and entity-centric knowledge structures in the real world, providing insights into how LMs internally organize and use entity information.

: https://github.com/masaki- sakata/entity-identification

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Language Models Represent Space and Time cites Locating and Editing Factual Associations in GPT
  • Language Models Represent Space and Time cites Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
  • Language Models Represent Space and Time cites Mapping Language Models to Grounded Conceptual Spaces
  • Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models cites Locating and Editing Factual Associations in GPT
  • Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models cites Mapping Language Models to Grounded Conceptual Spaces
  • ToMMeR - Efficient Entity Mention Detection from Large Language Models cites Locating and Editing Factual Associations in GPT
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. ToMMeR - Efficient Entity Mention Detection from Large Language ModelsVictor Morand, Nadi Tomeh, Josiane Mothe, Benjamin PiwowarskiACL 2026

Builds on18

  1. Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
  2. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of DataEmily M. Bender, Alexander KollerACL 2020 · 914 citations
  3. Language Models Represent Space and TimeWes Gurnee, Max TegmarkICLR 2024 · 303 citations
  4. Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language ModelsAsma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon et al.ICML 2024 · 197 citations
  5. Mapping Language Models to Grounded Conceptual SpacesRoma Patel, Ellie PavlickICLR 2022 · 197 citations

Related papers

  1. Emergence of a High-Dimensional Abstraction Phase in Language TransformersEmily Cheng, Diego Doimo, Corentin Kervadec, Iuri Macocco et al.ICLR 2025 · 1 citation
  2. Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language ModelsMichael Li, Nishant SubramaniACL 2026 · 3 citations
  3. NER Retriever: Zero-Shot Named Entity Retrieval with Type-Aware EmbeddingsOr Shachar, Uri Katz, Yoav Goldberg, Oren GlickmanEMNLP Findings 2025 · 1 citation
  4. Geometry of Decision Making in Language ModelsAbhinav Joshi, Divyanshu Bhatt, Ashutosh ModiNeurIPS 2025 · 12 citations
  5. LIEDER: Linguistically-Informed Evaluation for Discourse Entity RecognitionXiaomeng Zhu, Robert FrankACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines