SolEval: Benchmarking Large Language Models for Repository-level Solidity Smart Contract Generation

Large language models (LLMs) have transformed code generation. However, most existing approaches focus on mainstream languages such as Python and Java, neglecting the Solidity language, the predominant programming language for Ethereum smart contracts. Due to the lack of adequate benchmarks for Solidity, LLMs' ability to generate secure, costeffective smart contracts remains unexplored. To fill this gap, we construct SolEval, the first repository-level benchmark designed for Solidity smart contract generation, to evaluate the performance of LLMs on Solidity. Sol-Eval consists of 1,507 samples from 28 different repositories, covering 6 popular domains, providing LLMs with a comprehensive evaluation benchmark. Unlike the existing Solidity benchmark, SolEval not only includes complex function calls but also reflects the real-world complexity of the Ethereum ecosystem by incorporating Gas@k and Vul@k. We evaluate 16 LLMs on SolEval, and our results show that the best-performing LLM achieves only 26.29% Pass@10, highlighting substantial room for improvement in Solidity code generation by LLMs. Additionally, we conduct supervised fine-tuning (SFT) on Qwen-7B using SolEval, resulting in a significant performance improvement, with Pass@5 increasing from 16.67% to 58.33%, demonstrating the effectiveness of fine-tuning LLMs on our benchmark. We release our data and code at https: //github.com/pzy2000/SolEval . Stats: SolEval benchmark is made up of 1,125 samples. Task: Repository-level Code Generation. Metrics: ⑥ Pass@k and Compile@k, ⑦ Gas Fee and Vul. function ternary(bool condition, uint256 a, uint256 b) /** * @notice A branchless ternary function that returns one of two values based on a condition. * @param condition A boolean condition that determines which value to return. * @param a The value to return if the condition is true. * @param b The value to return if the condition is false. * @return The result of the ternary operation, either a or b. */