Lune

ACL Findings2024

CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow

Nathanaël Beau, Benoît Crabbé

Abstract

We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as Pandas, Numpy, and Regex, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,409 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models' strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLaMa 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Execution-Based Evaluation for Open-Domain Code Generation cites DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. GiFT: Gibbs Fine-Tuning for Code GenerationHaochen Li, Wanjin Feng, Xin Zhou, Zhiqi ShenACL 2025

Builds on3

  1. DS-1000: A Natural and Reliable Benchmark for Data Science Code GenerationYuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang et al.ICML 2023 · 504 citations
  2. Execution-Based Evaluation for Open-Domain Code GenerationZhiruo Wang, Shuyan Zhou, Daniel Fried, Graham NeubigEMNLP Findings 2023 · 20 citations
  3. The impact of lexical and grammatical processing on generating code from natural languageNathanaël Beau, Benoît CrabbéACL Findings 2022 · 18 citations

Related papers

  1. API Pack: A Massive Multi-Programming Language Dataset for API Call GenerationZhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen et al.ICLR 2025
  2. Contamination Means Overestimation? A Fine-Grained Empirical Study in Code IntelligenceZhen Yang, Hongyi Lin, Yifan He, Junqi Wang et al.ISSTA 2026
  3. Less Is More: On the Importance of Data Quality for Unit Test GenerationJunwei Zhang, Xing Hu, Shan Gao, Xin Xia et al.FSE 2025 · 2 citations
  4. An Empirical Study to Evaluate AIGC Detectors on Code ContentJian Wang, Shangqing Liu, Xiaofei Xie, Yi LiASE 2024 · 4 citations
  5. UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program TestingYifeng He, Jiabo Huang, Yuyang Rong, Yiwen Guo et al.ISSTA 2024 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines