EMNLP Findings2024
LLM as a metric critic for low resource relation identification
Zhe Yang, Yi Huang, Yaqin Chen, Xiaoting Wu, Junlan Feng, Chao Deng
Abstract
In extremely low resource relation identification scenario, small language models (SLMs) incline to overfit, which significantly diminishes their accuracy. Recently, large language models (LLMs) are gradually applied to classification tasks with converting original objective into the generation task via in-context learning. However, abundance of the classifier categories poses challenges in selecting demonstrations. Moreover, the mapping between category labels and textual descriptions requires expensive expert knowledge, thereby constraining the efficacy of in-context learning for LLMs. We uphold that SLM is optimal for handling classification tasks, and its shortcomings in the low resource setting can be mitigated by leveraging LLM. Hence, we propose a co-evolution strategy on SLM & LLM for relation identification. Specifically, LLM provides essential background knowledge to assist training process of the SLM classifier, while evaluation metrics from the classifier, in turn, offer valuable insights to refine the generation prompts of the LLM. We conduct experiments on several datasets which demonstrates preponderance of the proposed model.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Self-Refine: Iterative Refinement with Self-Feedback
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Self-Refine: Iterative Refinement with Self-Feedback cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Reflexion: language agents with verbal reinforcement learning cites Self-Refine: Iterative Refinement with Self-Feedback
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds on19
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
Related papers
- Multilingual LLMs are Better Cross-lingual In-context Learners with AlignmentEshaan Tanwar, Subhabrata Dutta, Manish Borthakur, Tanmoy ChakrabortyACL 2023 · 21 citations
- Combining Distantly Supervised Models with In Context Learning for Monolingual and Cross-Lingual Relation ExtractionVipul Kumar Rathore, Malik Hammad Faisal, Parag Singla, MausamACL 2026
- Generating Diverse Training Samples for Relation Extraction with Large Language ModelsZexuan Li, Hongliang Dai, Piji LiACL 2025
- Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation ExtractionXilai Ma, Jing Li, Min ZhangEMNLP Findings 2023 · 28 citations
- STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language ModelsMingyu Derek Ma, Xiaoxuan Wang, Po-Nien Kung, P. Jeffrey Brantingham et al.AAAI 2024 · 22 citations
