Lune

EMNLP Findings2023

Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis

Philip John Gorinski, Matthieu Zimmer, Gerasimos Lampouras, Derrick-Goh-Xin Deik, Ignacio Iacobacci

Abstract

The advent of large pre-trained language models in the domain of Code Synthesis has shown remarkable performance on various benchmarks, treating the problem of Code Generation in a fashion similar to Natural Language Generation, trained with a Language Modelling (LM) objective. In addition, the property of programming language code being precisely evaluable with respect to its semantics -through the use of Unit Tests to check its functional correctness -lends itself to using Reinforcement Learning (RL) as a further training paradigm. Previous work has shown that RL can be applied as such to improve models' coding capabilities; however, such RL-based methods rely on a reward signal based on defined Unit Tests, which are much harder to obtain compared to the huge crawled code datasets used in LM objectives. In this work, we present a novel approach to automatically obtain data consisting of function signatures and associated Unit Tests, suitable for RL training of Code Synthesis models. We also introduce a straightforward, simple yet effective Actor-Critic RL training scheme and show that it, in conjunction with automatically generated training data, leads to improvement of a pre-trained code language model's performance by up to 9.9% improvement over the original underlying code synthesis LM, and up to 4.3% over RL-based models trained with standard PPO or CodeRL.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning cites CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
  • CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis cites CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
  • CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis cites The Curious Case of Neural Text Degeneration
  • CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation cites Unsupervised Translation of Programming Languages
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  2. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and GenerationYue Wang, Weishi Wang, Shafiq R. Joty, Steven C. H. HoiEMNLP 2021 · 1,224 citations
  3. Unsupervised Translation of Programming LanguagesBaptiste Rozière, Marie-Anne Lachaux, Lowik Chanussot, Guillaume LampleNeurIPS 2020 · 606 citations
  4. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement LearningHung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese et al.NeurIPS 2022 · 571 citations
  5. CodeGen: An Open Large Language Model for Code with Multi-Turn Program SynthesisErik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu et al.ICLR 2023 · 234 citations

Related papers

  1. CodeRM-NT: Reward Model for Code RL without Unit TestsXiao Xia, Dan Zhang, Tianrui SunACL Findings 2026
  2. UnitCoder: Scalable Code Synthesis from Pre-training CorporaYichuan Ma, Yunfan Shao, Peiji Li, Demin Song et al.EMNLP 2025 · 2 citations
  3. ACECODER: Acing Coder RL via Automated Test-Case SynthesisHuaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie et al.ACL 2025 · 72 citations
  4. B-Coder: Value-Based Deep Reinforcement Learning for Program SynthesisZishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun et al.ICLR 2024 · 20 citations
  5. Co-Evolving LLM Coder and Unit Tester via Reinforcement LearningYinjie Wang, Ling Yang, Ye Tian, Ke Shen et al.NeurIPS 2025 · 56 citations
Dusk over the sea between two cliffs drawn in fine vertical lines