EMNLP Findings2024
MP-RNA: Unleashing Multi-species RNA Foundation Model via Calibrated Secondary Structure Prediction
Heng Yang, Ke Li
Abstract
RNA foundation models (FMs) have been extensively used to interpret genomic sequences and address a wide range of in-silico genomic tasks. However, current RNA FMs often overlook the incorporation of secondary structures in the pretraining of FMs, which impedes the effectiveness in various genomic tasks. To address this problem, we leverage filtered highfidelity structure annotations for structure pretraining to enhance the modeling ability of FMs in single nucleotide resolution tasks. Experimental evaluations across four comprehensive genomic benchmarks demonstrate that our FM (MP-RNA) consistently outperforms existing RNA FMs, achieving a 40% improvement in RNA secondary structure prediction and obtaining top-tier results on DNA genomic benchmarks even though it has not been pretrained on any DNA genome. We release the code and tutorials 1 and models to encourage further research to bridge the gap between in-silico predictions and biological reality.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on3
- HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide ResolutionEric Nguyen, Michael Poli, Marjan Faizi, Armin W. Thomas et al.NeurIPS 2023 · 574 citations
- Scaling Data-Constrained Language ModelsNiklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao et al.NeurIPS 2023 · 475 citations
- Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence ModelingYair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao et al.ICML 2024 · 195 citations
Related papers
- Bridging Sequence-Structure Alignment in RNA Foundation ModelsHeng Yang, Renzhi Chen, Ke LiAAAI 2025
- HELM: Hierarchical Encoding for mRNA Language ModelingMehdi Yazdani-Jahromi, Mangal Prakash, Tommaso Mansi, Artem Moskalev et al.ICLR 2025
- Beyond Sequence: Impact of Geometric Context for RNA Property PredictionJunjie Xu, Artem Moskalev, Tommaso Mansi, Mangal Prakash et al.ICLR 2025
- PriFold: Biological Priors Improve RNA Secondary Structure PredictionsChenchen Yang, Hao Wu, Tao Shen, Kai Zou et al.AAAI 2025
- BioToken and BioFM – Biologically-Informed Tokenization Enables Accurate and Efficient Genomic Foundation ModelsAleksandr Medvedev, Karthik Viswanathan, Praveenkumar Kanithi, Kirill Vishniakov et al.ICML 2026
