Lune

ACL Findings2024

Functional Overlap Reranking for Neural Code Generation

Hung To, Minh Nguyen, Nghi Bui

Abstract

Code Large Language Models (CodeLLMs) have ushered in a new era in code generation advancements. However, selecting the best code solutions from all possible CodeLLM outputs remains a challenge. Previous methods often overlooked the intricate functional similarities and interactions between solution clusters. We introduce SRank, a novel reranking strategy for selecting the best solutions from code generation, focusing on modeling the relationships between clusters of solutions. By quantifying the functional overlap between solution clusters, our approach provides a better ranking strategy for code solutions. Empirical results show that our method achieves remarkable results on the pass@1 score. For instance, on the Human-Eval benchmark, we achieve 69.66% in pass@1 with Codex002, 75.31% with Wiz-ardCoder, 53.99% with StarCoder, and 60.55% with CodeGen, surpassing state-of-the-art code generation reranking methods such as CodeT and Coder-Reviewer on the same CodeLLM by a significant margin (≈ 6.1% improvement on average). Even in scenarios with a limited number of sampled solutions and test cases, our approach demonstrates robustness and superiority, marking a new benchmark in code generation reranking. Our implementation can be found at https://github.com/ FSoft-AI4Code/SRank-CodeRanker.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries cites CodeBERT: A Pre-Trained Model for Programming and Natural Languages
  • ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries cites WizardCoder: Empowering Code Large Language Models with Evol-Instruct
  • CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation cites CodeBERT: A Pre-Trained Model for Programming and Natural Languages
  • Dynamic-Static Synergistic Selection Method for Candidate Code Solutions with Generated Test Cases cites WizardCoder: Empowering Code Large Language Models with Evol-Instruct
  • WizardCoder: Empowering Code Large Language Models with Evol-Instruct cites CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight ModelsMehrzad Samadi, Aleksander Ficek, Sean Narenthiran, Siddhartha Jain et al.ACL 2026 · 6 citations
  2. Dynamic-Static Synergistic Selection Method for Candidate Code Solutions with Generated Test CasesRenbiao Liu, Jiang-Tian Xue, Chao-Zeng Ma, Hui Sun et al.AAAI 2026 · 2 citations
  3. ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise QueriesTom Yuviler, Dana Drachsler-CohenAAAI 2026

Builds on16

  1. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  2. CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesZhangyin Feng, Daya Guo, Duyu Tang, Nan Duan et al.EMNLP Findings 2020 · 2,828 citations
  3. GraphCodeBERT: Pre-training Code Representations with Data FlowDaya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng et al.ICLR 2021 · 1,644 citations
  4. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 citations
  5. WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun et al.ICLR 2024 · 945 citations

Related papers

  1. Coder Reviewer Reranking for Code GenerationTianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis et al.ICML 2023 · 125 citations
  2. Lightweight reranking for language model generationsSiddhartha Jain, Xiaofei Ma, Anoop Deoras, Bing XiangACL 2024
  3. Language Ranker: A Lightweight Ranking framework for LLM DecodingChenheng Zhang, Tianqi Du, Jizhe Zhang, Mingqing Xiao et al.NeurIPS 2025 · 3 citations
  4. Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code CandidatesZhihong Sun, Yao Wan, Jia Li, Hongyu Zhang et al.ASE 2024 · 5 citations
  5. Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and BeyondMinh Le-Anh, Huyen Nguyen, Khanh An Tran, Nam Le Hai et al.FSE 2026
Dusk over the sea between two cliffs drawn in fine vertical lines