Lune

EMNLP Findings2025

EmByte: Decomposition and Compression Learning for Small yet Private NLP

Shenglan Li, Jia Xu, Mengjiao Zhang

Abstract

Recent breakthroughs in natural language processing (NLP) have come with escalating model sizes and computational costs, posing significant challenges for deployment in realtime and resource-constrained environments.

We introduce EMBYTE , a novel byte-level tokenization model that achieves substantial embedding compression while preserving NLP accuracy and enhancing privacy. At the core of EMBYTE is a new Decompose-and-Compress (DECOMP) learning strategy that decomposes subwords into fine-grained byte embeddings and then compresses them via neural projection. DECOMP enables EMBYTE to be shrunk down to any vocabulary size (e.g., 128 or 256), drastically reducing embedding parameter count by up to 94% compared to subword-based models without increasing sequence length or degrading performance. Moreover, EMBYTE is resilient to privacy threats such as gradient inversion attacks, due to its byte-level many-to-one mapping structure.

Empirical results on GLUE, machine translation, sentiment analysis, and language modeling tasks show that EMBYTE matches or surpasses the performance of significantly larger models, while offering improved efficiency. This makes EMBYTE a lightweight and generalizable NLP solution, well-suited for deployment in privacy-sensitive or low-resource environments.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • TinyBERT: Distilling BERT for Natural Language Understanding cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
  • MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices cites TinyBERT: Distilling BERT for Natural Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL ConferencesYusuke Sakai, Hidetaka Kamigaito, Taro WatanabeACL 2026 · 18 citations

Builds on11

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  3. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  4. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
  5. Large Language Models Can Be Strong Differentially Private LearnersXuechen Li, Florian Tramèr, Percy Liang, Tatsunori HashimotoICLR 2022 · 502 citations

Related papers

  1. SpaceByte: Towards Deleting Tokenization from Large Language ModelingKevin SlagleNeurIPS 2024 · 34 citations
  2. Neural Machine Translation with Byte-Level SubwordsChanghan Wang, Kyunghyun Cho, Jiatao GuAAAI 2020 · 213 citations
  3. ByteFlow: Language Modeling through Adaptive Byte Compression without a TokenizerChunyuan Deng, Sanket Lokegaonkar, Colin Lockard, Besnik Fetahu et al.ICLR 2026 · 1 citation
  4. Tensorized Embedding LayersOleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova et al.EMNLP Findings 2020 · 30 citations
  5. Learn Your Tokens: Word-Pooled Tokenization for Language ModelingAvijit Thawani, Saurabh Ghanekar, Xiaoyuan Zhu, Jay PujaraEMNLP Findings 2023 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines