Lune

ACL Findings2024

Code Needs Comments: Enhancing Code LLMs with Comment Augmentation

Demin Song, Honglin Guo, Yunhua Zhou, Shuhao Xing, Yudong Wang, Zifan Song, Wenwei Zhang, Qipeng Guo, Hang Yan, Xipeng Qiu, Dahua Lin

Abstract

The programming skill is one crucial ability for Large Language Models (LLMs), necessitating a deep understanding of programming languages (PLs) and their correlation with natural languages (NLs). We examine the impact of pre-training data on code-focused LLMs' performance by assessing the comment density as a measure of PL-NL alignment. Given the scarcity of code-comment aligned data in pre-training corpora, we introduce a novel data augmentation method that generates comments for existing code, coupled with a data filtering strategy that filters out code data poorly correlated with natural language. We conducted experiments on three code-focused LLMs and observed consistent improvements in performance on two widely-used programming skill benchmarks. Notably, the model trained on the augmented data outperformed both the model used for generating comments and the model further trained on the data without augmentation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages cites CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source DataZifan Song, Yudong Wang, Wenwei Zhang, Kuikun Liu et al.NeurIPS 2024 · 8 citations
  2. Inside Out: Uncovering How Comment Internalization Steers LLMs for Better or WorseAaron Imani, Mohammad Moshirpour, Iftekhar AhmedICSE 2026
  3. Black-Box Adversarial Attacks on LLM-Based Code CompletionSlobodan Jenko, Niels Mündler, Jingxuan He, Mark Vero et al.ICML 2025
  4. CritiQ: Mining Data Quality Criteria from Human PreferencesHonglin Guo, Kai Lv, Qipeng Guo, Tianyi Liang et al.ACL 2025
  5. Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled PreferencesMohammad Saqib Hasan, Saikat Chakraborty, Santu Karmaker, Niranjan BalasubramanianACL 2025

Builds on6

  1. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 citations
  2. Self-Instruct: Aligning Language Models with Self-Generated InstructionsYizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu et al.ACL 2023 · 540 citations
  3. Misleading Authorship Attribution of Source Code using Adversarial LearningErwin Quiring, Alwin Maier, Konrad RieckUSENIX Security 2019 · 123 citations
  4. ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming LanguagesYekun Chai, Shuohuan Wang, Chao Pang, Yu Sun et al.ACL Findings 2023 · 18 citations
  5. Achieving Human Parity in Content-Grounded Datasets GenerationAsaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv et al.ICLR 2024 · 9 citations

Related papers

  1. Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context LearningMingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang et al.ICSE 2024 · 124 citations
  2. Automated Program Repair in the Era of Large Pre-trained Language ModelsChunqiu Steven Xia, Yuxiang Wei, Lingming ZhangICSE 2023 · 321 citations
  3. Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma et al.FSE 2024 · 8 citations
  4. CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain SettingsDaniil Orel, Dilshod Azizov, Preslav NakovACL Findings 2025 · 5 citations
  5. Better Language Models of Code through Self-ImprovementHung Quoc To, Nghi D. Q. Bui, Jin L. C. Guo, Tien N. NguyenACL Findings 2023 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines