S*: Test Time Scaling for Code Generation

Increasing test-time compute for LLMs shows promise across domains but remains underexplored in code generation, despite extensive study in math. In this paper, we propose S * , the first hybrid test-time scaling framework that substantially improves the coverage and selection accuracy of generated code. S * augments the existing parallel scaling approach with sequential scaling to further increase the performance. It further leverages a novel selection mechanism that adaptively generates distinguishing inputs for pairwise comparison, combined with execution-grounded information to robustly identify correct solutions.

We evaluate S * across 12 Large Language Models and Large Reasoning Models and show that: (1) S * consistently improves performance across model families and sizes, enabling a 3B model to outperform GPT-4omini; (2) S * enables non-reasoning models to surpass reasoning models-GPT-4o-mini with S * outperforms o1-preview by 3.7% on Live-CodeBench; (3) S * further boosts state-of-theart reasoning models-DeepSeek-R1-Distill-Qwen-32B with S * achieves 85.7% on Live-CodeBench, approaching o1 (high) at 88.5%. Codes, model generations and intermediate experiments results are available under https: //github.com/NovaSky-AI/SkyThought.