ACL Findings2024
CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow
Abstract
We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as Pandas, Numpy, and Regex, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,409 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models' strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLaMa 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on3
- DS-1000: A Natural and Reliable Benchmark for Data Science Code GenerationYuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang et al.ICML 2023 · 504 citations
- Execution-Based Evaluation for Open-Domain Code GenerationZhiruo Wang, Shuyan Zhou, Daniel Fried, Graham NeubigEMNLP Findings 2023 · 20 citations
- The impact of lexical and grammatical processing on generating code from natural languageNathanaël Beau, Benoît CrabbéACL Findings 2022 · 18 citations
Related papers
- API Pack: A Massive Multi-Programming Language Dataset for API Call GenerationZhen Guo, Adriana Meza Soria, Wei Sun, Yikang Shen et al.ICLR 2025
- Contamination Means Overestimation? A Fine-Grained Empirical Study in Code IntelligenceZhen Yang, Hongyi Lin, Yifan He, Junqi Wang et al.ISSTA 2026
- Less Is More: On the Importance of Data Quality for Unit Test GenerationJunwei Zhang, Xing Hu, Shan Gao, Xin Xia et al.FSE 2025 · 2 citations
- An Empirical Study to Evaluate AIGC Detectors on Code ContentJian Wang, Shangqing Liu, Xiaofei Xie, Yi LiASE 2024 · 4 citations
- UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program TestingYifeng He, Jiabo Huang, Yuyang Rong, Yiwen Guo et al.ISSTA 2024 · 10 citations
