Lune

EMNLP Findings2020

Will it Unblend?

Yuval Pinter, Cassandra L. Jacobs, Jacob Eisenstein

Abstract

Natural language processing systems often struggle with out-of-vocabulary (OOV) terms, which do not appear in training data. Blends, such as innoventor, are one particularly challenging class of OOV, as they are formed by fusing together two or more bases that relate to the intended meaning in unpredictable manners and degrees. In this work, we run experiments on a novel dataset of English OOV blends to quantify the difficulty of interpreting the meanings of blends by large-scale contextual language models such as BERT. We first show that BERT's processing of these blends does not fully access the component meanings, leaving their contextual representations semantically impoverished. We find this is mostly due to the loss of characters resulting from blend formation. Then, we assess how easily different models can recognize the structure and recover the origin of blends, and find that context-aware embedding systems outperform character-level and contextfree embeddings, although their results are still far from satisfactory.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words cites Byte Pair Encoding is Suboptimal for Language Model Pretraining
  • Dynamic Contextualized Word Embeddings cites Analysing Lexical Semantic Change with Contextualised Word Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Evaluating Lexical Proficiency in Neural Language ModelsCristiano Ciaccio, Alessio Miaschi, Felice Dell'OrlettaACL 2025
  2. Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex WordsValentin Hofmann, Janet B. Pierrehumbert, Hinrich SchützeACL 2021
  3. Dynamic Contextualized Word EmbeddingsValentin Hofmann, Janet B. Pierrehumbert, Hinrich SchützeACL 2021

Builds on3

  1. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
  2. Byte Pair Encoding is Suboptimal for Language Model PretrainingKaj Bostrom, Greg DurrettEMNLP Findings 2020 · 180 citations
  3. Analysing Lexical Semantic Change with Contextualised Word RepresentationsMario Giulianelli, Marco Del Tredici, Raquel FernándezACL 2020 · 118 citations

Related papers

  1. Imputing Out-of-Vocabulary Embeddings with LOVE Makes LanguageModels Robust with Little CostLihu Chen, Gaël Varoquaux, Fabian M. SuchanekACL 2022
  2. How do Transformer Embeddings Represent Compositions? A Functional AnalysisAishik Nagar, Ishaan Singh Rawal, Mansi Dhanania, Cheston TanACL Findings 2025
  3. Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive MimickingTimo Schick, Hinrich SchützeAAAI 2020 · 106 citations
  4. Robust Backed-off Estimation of Out-of-Vocabulary EmbeddingsNobukazu Fukuda, Naoki Yoshinaga, Masaru KitsuregawaEMNLP Findings 2020 · 15 citations
  5. Incorporating BERT into Neural Machine TranslationJinhua Zhu, Yingce Xia, Lijun Wu, Di He et al.ICLR 2020 · 391 citations
Dusk over the sea between two cliffs drawn in fine vertical lines