Lune

EMNLP2024Top-tier venue

How Do Your Code LLMs perform? Empowering Code Instruction Tuning with Really Good Data

Yejie Wang, Keqing He, Dayuan Fu, Zhuoma Gongque, Heyang Xu, Yanxu Chen, Zhexu Wang, Yujia Fu, Guanting Dong, Muxi Diao, Jingang Wang, Mengdi Zhang

Abstract

Recently, there has been a growing interest in studying how to construct better code instruction tuning data. However, we observe Code models trained with these datasets exhibit high performance on HumanEval but perform worse on other benchmarks such as Live-CodeBench. Upon further investigation, we find that many datasets suffer from severe data leakage. After cleaning up most of the leaked data, some well-known high-quality datasets perform poorly. This discovery reveals a new challenge: identifying which dataset genuinely qualify as high-quality code instruction data. To address this, we propose an efficient code data pruning strategy for selecting good samples. Our approach is based on three dimensions: instruction complexity, response quality, and instruction diversity. Based on our selected data, we present XCoder 1 , a family of models finetuned from LLaMA3. Our experiments show XCoder achieves new state-ofthe-art performance using fewer training data, which verify the effectiveness of our data strategy. Moreover, we perform a comprehensive analysis on the data composition and find existing code datasets have different characteristics according to their construction methods, which provide new insights for future code LLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement cites CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
  • LAMDAS: LLM as an Implicit Classifier for Domain-specific Data Selection cites What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
  • TRACE: Trajectory-based Activation Change Estimation for Task-specific Data Selection cites What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
  • What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning cites #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models
  • Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences cites OctoPack: Instruction Tuning Code Large Language Models
  • Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences cites CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. LAMDAS: LLM as an Implicit Classifier for Domain-specific Data SelectionJian Wu, Hang Yu, Bingchang Liu, Wenjie Yang et al.AAAI 2026 · 1 citation
  2. TRACE: Trajectory-based Activation Change Estimation for Task-specific Data SelectionYe He, Shangzhan Li, Yuxin Zhou, Qi ShiAAAI 2026
  3. Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled PreferencesMohammad Saqib Hasan, Saikat Chakraborty, Santu Karmaker, Niranjan BalasubramanianACL 2025

Builds on8

  1. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement LearningHung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese et al.NeurIPS 2022 · 571 citations
  2. What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction TuningWei Liu, Weihao Zeng, Keqing He, Yong Jiang et al.ICLR 2024 · 369 citations
  3. OctoPack: Instruction Tuning Code Large Language ModelsNiklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng et al.ICLR 2024 · 203 citations
  4. #InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language ModelsKeming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin et al.ICLR 2024 · 119 citations
  5. OpenCodeInterpreter: Integrating Code Generation with Execution and RefinementTianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu et al.ACL Findings 2024 · 46 citations

Related papers

  1. WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction TuningZhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang et al.ACL 2024
  2. SelfCodeAlign: Self-Alignment for Code GenerationYuxiang Wei, Federico Cassano, Jiawei Liu, Yifeng Ding et al.NeurIPS 2024 · 79 citations
  3. WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun et al.ICLR 2024 · 945 citations
  4. From Selection to Refinement: Iterative Optimization for Instruction DataHang Hu, Ziyan Liu, Rujie Wen, Ruihui Hou et al.ACL 2026
  5. KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for CodingZhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou et al.ACL Findings 2025 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines