ACL Findings2025
Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou
Abstract
Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios. As for the cross-tokenizer KD, the differences in the tokenizers give rise to two fundamental challenges: (1) sequence misalignment caused by divergent tokenization strategies, and (2) mismatched vocabulary size and composition. While existing probability-matching methods attempt to address these issues, their efficacy remains limited due to suboptimal alignment in both the sequence and vocabulary aspects. To overcome these limitations, we propose Contextual Dynamic Mapping (CDM), a novel crosstokenizer distillation framework that employs contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping. We evaluated the effectiveness of our approach across five advanced and widely-used model families (i.e, LLama3, Phi3, Gemma2, OPT and Qwen2), which were configured into three distinct teacher-student pairs. Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks, including instruction-following, code generation and math. Notably, our analysis reveals that combining conventional same-tokenizer distillation and cross-tokenizer distillation through CDM yields further performance improvements. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Entropy-aware Span-Constrained Optimal Transport for Robust Cross-Tokenizer Knowledge Distillation cites Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
- Entropy-aware Span-Constrained Optimal Transport for Robust Cross-Tokenizer Knowledge Distillation cites Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Entropy-aware Span-Constrained Optimal Transport for Robust Cross-Tokenizer Knowledge Distillation cites Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
- MiniLLM: Knowledge Distillation of Large Language Models cites Self-Instruct: Aligning Language Models with Self-Generated Instructions
- MiniLLM: Knowledge Distillation of Large Language Models cites Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
- SRA: Span Representation Alignment for Large Language Model Distillation cites MiniLLM: Knowledge Distillation of Large Language Models
- SRA: Span Representation Alignment for Large Language Model Distillation cites Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
- SRA: Span Representation Alignment for Large Language Model Distillation cites Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Knowledge Distillation for Large Language Models through Residual Learning cites MiniLLM: Knowledge Distillation of Large Language Models
- Knowledge Distillation for Large Language Models through Residual Learning cites Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
- Knowledge Distillation for Large Language Models through Residual Learning cites Self-Instruct: Aligning Language Models with Self-Generated Instructions
- Knowledge Distillation for Large Language Models through Residual Learning cites Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
Cited by top-tier papers7
- ProxyThinker: Test-Time Guidance through Small Visual ReasonersZilin Xiao, Jaywon Koo, Siru Ouyang, Jefferson Hernandez et al.ICLR 2026 · 8 citations
- Cross-Tokenizer Likelihood Scoring Algorithms for Language Model DistillationBuu Phan, Ashish Khisti, Karen UllrichICLR 2026 · 5 citations
- Knowledge Distillation for Large Language Models through Residual LearningThinh On, Hengzhi Pei, Leonard Lausen, George KarypisICLR 2026 · 5 citations
- SRA: Span Representation Alignment for Large Language Model DistillationQuoc Phong Dao, Hoang Son Nguyen, Pham Khanh Chi, Tung Nguyen et al.ACL 2026 · 1 citation
- Entropy-aware Span-Constrained Optimal Transport for Robust Cross-Tokenizer Knowledge DistillationZhi-Ping Liu, Simiao Li, Wei Li, Hanting Chen et al.ICML 2026
Builds on11
- Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming ZhangNeurIPS 2023 · 2,317 citations
- Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu et al.ACL 2023 · 540 citations
- Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksYizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi et al.EMNLP 2022 · 238 citations
- MiniLLM: Knowledge Distillation of Large Language ModelsYuxian Gu, Li Dong, Furu Wei, Minlie HuangICLR 2024 · 95 citations
- Unnatural Instructions: Tuning Language Models with (Almost) No Human LaborOr Honovich, Thomas Scialom, Omer Levy, Timo SchickACL 2023 · 92 citations
Related papers
- EMO: Embedding Model Distillation via Intra-Model Relation and Optimal Transport AlignmentsMinh-Phuc Truong, Hai An Vu, Tu Vu, Nguyen Thi Ngoc Diep et al.EMNLP 2025
- MCW-KD: Multi-Cost Wasserstein Knowledge Distillation for Large Language ModelsHoang Tran Vuong, Tue Le, Quyen Tran, Linh Ngo Van et al.AAAI 2026
- Dual-Space Knowledge Distillation for Large Language ModelsSongming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen et al.EMNLP 2024 · 3 citations
- Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language ModelsXiao Cui, Mo Zhu, Yulei Qin, Liang Xie et al.AAAI 2025 · 31 citations
- Towards Efficient Pre-Trained Language Model via Feature Correlation DistillationKun Huang, Xin Guo, Meng WangNeurIPS 2023 · 8 citations
