EMNLP Findings2020
Research Replication Prediction Using Weakly Supervised Learning
Abstract
Knowing whether a published research result can be replicated is important. Carrying out direct replication of published research incurs a high cost. There are efforts tried to use machine learning aided methods to predict scientific claims' replicability. However, existing machine learning aided approaches use only hand-extracted statistics features such as p-value, sample size, etc. without utilizing research papers' text information and train only on a very small size of annotated data without making the most use of a large number of unlabeled articles. Therefore, it is desirable to develop effective machine learning aided automatic methods which can automatically extract text information as features so that we can benefit from Natural Language Processing techniques. Besides, we aim for an approach that benefits from both labeled and the large number of unlabeled data. In this paper, we propose two weakly supervised learning approaches that use automatically extracted text information of research papers to improve the prediction accuracy of research replication using both labeled and unlabeled datasets. Our experiments over real-world datasets show that our approaches obtain much better prediction performance compared to the supervised models utilizing only statistic features and a small size of labeled dataset. Further, we are able to achieve an accuracy of 75.76% for predicting the replicability of research.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features cites To Aggregate or Not? Learning with Separate Noisy Labels
- To Aggregate or Not? Learning with Separate Noisy Labels cites Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features
- Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features cites Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates
- Robust Learning from Noisily Labeled Long-Tailed Data via Fairness Regularizer cites Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality Features
- Robust Learning from Noisily Labeled Long-Tailed Data via Fairness Regularizer cites Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates
- To Aggregate or Not? Learning with Separate Noisy Labels cites Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates
Cited by top-tier papers3
- Beyond Images: Label Noise Transition Matrix Estimation for Tasks with Lower-Quality FeaturesZhaowei Zhu, Jialu Wang, Yang LiuICML 2022 · 43 citations
- To Aggregate or Not? Learning with Separate Noisy LabelsJiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid et al.KDD 2023 · 21 citations
- Robust Learning from Noisily Labeled Long-Tailed Data via Fairness RegularizerJiaheng Wei, Zhaowei Zhu, Gang Niu, Tongliang Liu et al.AAAI 2026
Builds on2
Related papers
- FastClass: A Time-Efficient Approach to Weakly-Supervised Text ClassificationTingyu Xia, Yue Wang, Yuan Tian, Yi ChangEMNLP 2022 · 1 citation
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning MethodsLuisa Werner, Nabil Layaïda, Pierre Genevès, Jérôme Euzenat et al.AAAI 2024 · 2 citations
- The Unreasonable Effectiveness of Open Science in AI: A Replication StudyOdd Erik Gundersen, Odd Cappelen, Martin Mølnå, Nicklas Grimstad NilsenAAAI 2025 · 8 citations
- ReactIE: Enhancing Chemical Reaction Extraction with Weak SupervisionMing Zhong, Siru Ouyang, Minhao Jiang, Vivian Hu et al.ACL Findings 2023 · 5 citations
- Unsupervised Readability Assessment via Learning from Weak Readability SignalsYuliang Liu, Zhiwei Jiang, Yafeng Yin, Cong Wang et al.SIGIR 2023 · 3 citations
