Rethinking the generalization of drug target affinity prediction algorithms via similarity aware evaluation
Chenbin Zhang, Zhiqiang Hu, Chuchu Jiang, Wen Chen, Jie Xu, Shaoting Zhang
Abstract
Drug-target binding affinity prediction is a fundamental task for drug discovery. It has been extensively explored in literature and promising results are reported. However, in this paper, we demonstrate that the results may be misleading and cannot be well generalized to real practice. The core observation is that the canonical randomized split of a test set in conventional evaluation leaves the test set dominated by samples with high similarity to the training set. The performance of models is severely degraded on samples with lower similarity to the training set but the drawback is highly overlooked in current evaluation. As a result, the performance can hardly be trusted when the model meets low-similarity samples in real practice. To address this problem, we propose a framework of similarity aware evaluation in which a novel split methodology is proposed to adapt to any desired distribution. This is achieved by a formulation of optimization problems which are approximately and efficiently solved by gradient descent. We perform extensive experiments across five representative methods in four datasets for two typical target evaluations and compare them with various counterpart methods. Results demonstrate that the proposed split methodology can significantly better fit desired distributions and guide the development of models. Code is released at https://github.com/Amshoreline/SAE/tree/main.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- Uni-Mol: A Universal 3D Molecular Representation Learning FrameworkGengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng et al.ICLR 2023 · 254 citations
- 3D Equivariant Diffusion for Target-Aware Molecule Generation and Affinity PredictionJiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su et al.ICLR 2023 · 79 citations
- Rethinking Tokenizer and Decoder in Masked Graph Modeling for MoleculesZhiyuan Liu, Yaorui Shi, An Zhang, Enzhi Zhang et al.NeurIPS 2023 · 71 citations
- Meta-learning Adaptive Deep Kernel Gaussian Processes for Molecular Property PredictionWenlin Chen, Austin Tripp, José Miguel Hernández-LobatoICLR 2023 · 8 citations
Related papers
- A new framework for evaluating model out-of-distribution generalisation for the biochemical domainRaúl Fernández-Díaz, Hoang Thanh Lam, Vanessa López, Denis C. ShieldsICLR 2025 · 5 citations
- DrugOOD: Out-of-Distribution Dataset Curator and Benchmark for AI-Aided Drug Discovery - a Focus on Affinity Prediction Problems with Noise AnnotationsYuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bingzhe Wu et al.AAAI 2023 · 65 citations
- Exploring Chemical Space with Score-based Out-of-distribution GenerationSeul Lee, Jaehyeong Jo, Sung Ju HwangICML 2023 · 110 citations
- EquiBind: Geometric Deep Learning for Drug Binding Structure PredictionHannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay et al.ICML 2022 · 360 citations
- DiSCO: Diffusion Schrödinger Bridge for Molecular Conformer OptimizationDanyeong Lee, Dohoon Lee, Dongmin Bang, Sun KimAAAI 2024 · 4 citations
