Environment-Aware Code Generation: How far are We?

Recent progress in large language models (LLMs) has led to impressive code generation capabilities. However, existing evaluations of LLMs primarily focus on generating isolated, small-scale code units (e.g., single functions or statements) under default or unspecified software environments. As a result, it remains unclear whether LLMs can reliably generate executable code tailored to specific user environments. To fill this knowledge gap, we make the first systematic study of Environment-Aware Code Generation (EACG), which requires generating code that is both functionally correct and directly executable under arbitrary software configurations. To support realistic evaluation, we introduce VersiBCB, a benchmark featuring multi-package, executable-verified, and deprecation-aware, reflecting complex and evolving software environments that are often overlooked in prior datasets. Building on this benchmark, we explore three orthogonal adaptation axes: data, parameters, and cache, and further develop representative strategies for each. Our results reveal that existing LLMs struggle with environment-specific code generation, but our adaptation strategies yield improvements in environment compatibility and executability. These findings highlight critical challenges and opportunities for deploying LLMs in practical software engineering workflows.

• Software and its engineering → Software development techniques; • Computing methodologies → Natural language processing.