A Large Scale Study of AI-based Binary Function Similarity Detection Techniques for Security Researchers and Practitioners
Jingyi Shi, Yufeng Chen, Yang Xiao, Yuekang Li, Zhengzi Xu, Sihao Qiu, Chi Zhang, Keyu Qi, Yeting Li, Xingchu Chen, Yanyan Zou, Yang Liu, Wei Huo
Abstract
Binary Function Similarity Detection (BFSD) is a foundational technique in software security, underpinning a wide range of applications including vulnerability detection, malware analysis. Recent advances in AI-based BFSD tools have led to significant performance improvements. However, existing evaluations of these tools suffer from three key limitations: a lack of in-depth analysis of performance-influencing factors, an absence of realistic application analysis, and reliance on small-scale or low-quality datasets. In this paper, we present the first large-scale empirical study of AI-based BFSD tools to address these gaps. We construct two high-quality and diverse datasets: BINATLAS, comprising 12,453 binaries and over 7 million functions for capability evaluation; and BINARES, containing 12,291 binaries and 54 real-world 1-day vulnerabilities for evaluating vulnerability detection performance in practical IoT firmware settings. Using these datasets, we evaluate nine representative BFSD tools, analyze the challenges and limitations of existing BFSD tools, and investigate the consistency among BFSD tools. We also propose an actionable strategy for combining BFSD tools to enhance overall performance (an improvement of 13.4%). Our study not only advances the practical adoption of BFSD tools but also provides valuable resources and insights to guide future research in scalable and automated binary similarity detection.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Asm2Vec: Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler Optimization cites Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection
- Asm2Vec: Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler Optimization cites discovRE: Efficient Cross-Architecture Identification of Bugs in Binary Code
- Asm2Vec: Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler Optimization cites Scalable Graph-based Bug Search for Firmware Images
- Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection cites discovRE: Efficient Cross-Architecture Identification of Bugs in Binary Code
- Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection cites Scalable Graph-based Bug Search for Firmware Images
- Scalable Graph-based Bug Search for Firmware Images cites discovRE: Efficient Cross-Architecture Identification of Bugs in Binary Code
Builds on25
- Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity DetectionXiaojun Xu, Chang Liu, Qian Feng, Heng Yin et al.CCS 2017 · 682 citations
- Scalable Graph-based Bug Search for Firmware ImagesQian Feng, Rundong Zhou, Chengcheng Xu, Yao Cheng et al.CCS 2016 · 456 citations
- Asm2Vec: Boosting Static Representation Robustness for Binary Clone Search against Code Obfuscation and Compiler OptimizationSteven H. H. Ding, Benjamin C. M. Fung, Philippe CharlandS&P 2019 · 447 citations
- discovRE: Efficient Cross-Architecture Identification of Bugs in Binary CodeSebastian Eschweiler, Khaled Yakdan, Elmar Gerhards-PadillaNDSS 2016 · 342 citations
- Order Matters: Semantic-Aware Neural Networks for Binary Code Similarity DetectionZeping Yu, Rui Cao, Qiyi Tang, Sen Nie et al.AAAI 2020 · 265 citations
Related papers
- Understanding Binary Code Similarity for Real-World Vulnerability Detection: A Large-Scale Empirical StudyJingdong Guo, Chaopeng Dong, Yimo Ren, Siyuan Li et al.FSE 2026
- CEBin: A Cost-Effective Framework for Large-Scale Binary Code Similarity DetectionHao Wang, Zeyu Gao, Chao Zhang, Mingyang Sun et al.ISSTA 2024 · 21 citations
- Revisiting Graph Representations for ML-Based Binary Code Similarity Detection: A Systematic StudyTengteng Yang, Yikun Hu, Jican Zhang, Lei Xue et al.ISSTA 2026
- KEENHash: Hashing Programs into Function-Aware Embeddings for Large-Scale Binary Code Similarity AnalysisZhijie Liu, Qiyi Tang, Sen Nie, Shi Wu et al.ISSTA 2025 · 1 citation
- A Comprehensive Empirical Analysis of Patch Presence Testing: Capabilities, Limitations, and Paths ForwardXiaobei Zhang, Yaowen Zheng, Wu Luo, Shijun Zhao et al.ISSTA 2026
