Lune

EMNLP Findings2024

Evolutionary Contrastive Distillation for Language Model Alignment

Julian Katz-Samuels, Zheng Li, Hyokun Yun, Priyanka Nigam, Yi Xu, Vaclav Petricek, Bing Yin, Trishul Chilimbi

Abstract

The ability of large language models (LLMs) to execute complex instructions is essential for their real-world applications. However, several recent studies indicate that LLMs struggle with challenging instructions (Zhou et al., 2023;Qin et al., 2024; Jiang et al., 2023b). In this paper, we propose Evolutionary Contrastive Distillation (ECD), a novel method for generating high-quality synthetic preference data designed to enhance the complex instructionfollowing capability of language models. ECD generates data that specifically illustrates the difference between a response that successfully follows a set of complex instructions and a response that is high-quality, but nevertheless makes some subtle mistakes. This is done by prompting LLMs to progressively evolve simple instructions into more complex instructions. When an instruction is made more complex, the original successful response mostly meets the new requirements but misses one or two, thus becoming a "hard negative" example for the new instruction. By pairing a good response with such a hard negative response, and employing contrastive learning algorithms such as DPO (Rafailov et al., 2023), we improve language models' ability to follow complex instructions. Empirically, we observe that our method yields a 7B model that exceeds the complex instruction-following performance of current state-of-the-art (SOTA) 7B models and is competitive even with open-source 70B models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • UltraIF: Advancing Instruction Following from the Wild cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • MindfulAgents: Personalizing Mindfulness Meditation via an Expert-Aligned Multi-Agent System cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Recontextualization Mitigates Specification Gaming Without Modifying the SpecificationAriana Azarbal, Victor Gillioz, Vladimir Ivanov, Bryce Woodworth et al.ICML 2026 · 11 citations
  2. UltraIF: Advancing Instruction Following from the WildKaikai An, Li Sheng, Ganqu Cui, Shuzheng Si et al.EMNLP 2025 · 1 citation
  3. MindfulAgents: Personalizing Mindfulness Meditation via an Expert-Aligned Multi-Agent SystemMengyuan Millie Wu, Zhihan Jiang, Yuang Fan, Richard Feng et al.CHI 2026 · 1 citation
  4. CLAOCS-TX: Cross-Lingual Triplet Extraction with Aspect-Opinion-Aware Code-Switched Prompting and LLM-Guided Contrastive DistillationLipika Dewangan, Chandresh Kumar MauryaACL 2026

Builds on24

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
  4. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
  5. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations

Related papers

  1. RLCD: Reinforcement Learning from Contrastive Distillation for LM AlignmentKevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng et al.ICLR 2024 · 37 citations
  2. DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMsJongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding et al.ICML 2025
  3. Self-Boosting Large Language Models with Synthetic Preference DataQingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui et al.ICLR 2025
  4. IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference OptimizationXinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang et al.ACL 2025 · 29 citations
  5. Better Alignment with Instruction Back-and-Forth TranslationThao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt et al.EMNLP Findings 2024 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines