KGARevion: An AI Agent for Knowledge-Intensive Biomedical QA
Xiaorui Su, Yibo Wang, Shanghua Gao, Xiaolong Liu, Valentina Giunchiglia, Djork-Arné Clevert, Marinka Zitnik
Abstract
Biomedical reasoning integrates structured, codified knowledge with tacit, experience-driven insights. Depending on the context, quantity, and nature of available evidence, researchers and clinicians use diverse strategies, including rule-based, prototype-based, and case-based reasoning. Effective medical AI models must handle this complexity while ensuring reliability and adaptability. We introduce KGAREVION, a knowledge graph-based agent that answers knowledge-intensive questions. Upon receiving a query, KGAREVION generates relevant triplets by leveraging the latent knowledge embedded in a large language model. It then verifies these triplets against a grounded knowledge graph, filtering out errors and retaining only accurate, contextually relevant information for the final answer. This multi-step process strengthens reasoning, adapts to different models of medical inference, and outperforms retrieval-augmented generationbased approaches that lack effective verification mechanisms. Evaluations on medical QA benchmarks show that KGAREVION improves accuracy by over 5.2% over 15 models in handling complex medical queries. To further assess its effectiveness, we curated three new medical QA datasets with varying levels of semantic complexity, where KGAREVION improved accuracy by 10.4%. The agent integrates with different LLMs and biomedical knowledge graphs for broad applicability across knowledge-intensive tasks. We evaluated KGAREVION on AfriMed-QA, a newly introduced dataset focused on African healthcare, demonstrating its strong zero-shot generalization to underrepresented medical contexts. Present work. We introduce KGAREVION, a KG-based LLM agent designed for complex biomedical QA that integrates the non-codified knowledge of LLMs with the structured, codified knowledge found in KGs. As illustrated in Fig. 2 , KGAREVION executes four key actions to ensure accurate and context-aware biomedical reasoning. First, it prompts the LLM to generate relevant triplets based on the input question. To effectively leverage structured KG data, KGAREVION fine-tunes the LLM on a KG completion task, incorporating pre-trained structural embeddings of triplets as prefix tokens. The fine-tuned model then evaluates the correctness of the generated triplets. Next, KGAREVION performs a 'Revise' action to correct erroneous triplets, refining the knowledge base before selecting the final answer. Given the complexity of medical reasoning, KGAREVION adaptively chooses the most appropriate reasoning approach for each question, allowing for nuanced and context-aware QA. This flexibility enables KGAREVION to handle both multi-choice and openended QA tasks. Our key contributions include: 1 ⃝ Developing KGAREVION, a versatile KG agent that dynamically adjusts reasoning strategies, achieving a 6.75% improvement over 15 baseline models in seven datasets, including three challenging newly curated benchmarks. 2 ⃝ Demonstrating that grounding through generated triplets significantly enhances KGAREVION's capabilities across multiple KGs. 3 ⃝ Showing that KGAREVION effectively answers complex, knowledge-intensive medical queries in both multi-choice and open-ended QA formats. 4 ⃝ Evaluating KGAREVION on African healthcare datasets: We benchmark KGAREVION on AfriMed-QA, a newly introduced dataset focused on African healthcare. The results highlight KGAREVION's strong zero-shot generalization, demonstrating its ability to reason effectively in underrepresented medical contexts. 5 ⃝ Analyzing robustness to input variations: We analyze KGAREVION's sensitivity to changes in question structure, answer ordering, and answer relabeling. Unlike LLMs, which exhibit high variance when answer choices are reordered, KGAREVION maintains stable performance, demonstrating its stronger robustness in real-world settings. KGAREVION is available at https://github.com/mims-harvard/KGARevion .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Reflexion: language agents with verbal reinforcement learning cites Toolformer: Language Models Can Teach Themselves to Use Tools
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- MedLA: A Logic-Driven Multi-Agent Framework for Complex Medical Reasoning with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
Cited by top-tier papers3
- MedLA: A Logic-Driven Multi-Agent Framework for Complex Medical Reasoning with Large Language ModelsSiqi Ma, Jiajie Huang, Fan Zhang, Jinlin Wu et al.AAAI 2026 · 10 citations
- KnowGuard: Knowledge-Driven Abstention for Multi-Round Clinical ReasoningXilin Dang, Kexin Chen, Xiaorui Su, Ayush Noori et al.ICLR 2026 · 6 citations
- MediEval: A Unified Medical Benchmark for Patient-Contextual and Knowledge-Grounded Reasoning in LLMsZhan Qu, Michael FärberACL 2026 · 1 citation
Builds on19
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu et al.NeurIPS 2023 · 5,989 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
Related papers
- Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community RetrievalPengcheng Jiang, Cao Xiao, Minhao Jiang, Parminder Bhatia et al.ICLR 2025
- PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question AnsweringYiqing Zhang, Xiaozhong Liu, Fabricio MuraiACL 2026
- KG-Agent: An Efficient Autonomous Agent Framework for Complex Reasoning over Knowledge GraphJinhao Jiang, Kun Zhou, Xin Zhao, Yang Song et al.ACL 2025
- ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical ReasoningYu Sun, Xingyu Qian, Weiwen Xu, Hao Zhang et al.EMNLP 2025 · 1 citation
- Iterative Knowledge Graph Refinement and Integration for Medical Question AnsweringHaochen Zou, Yongli WangACL Findings 2026
