EMNLP Findings2023
Definitions Matter: Guiding GPT for Multi-label Classification
Youri Peskine, Damir Korencic, Ivan Grubisic, Paolo Papotti, Raphaël Troncy, Paolo Rosso
摘要
Large language models have recently risen in popularity due to their ability to perform many natural language tasks without requiring any fine-tuning. In this work, we focus on two novel ideas: (1) generating definitions from examples and using them for zero-shot classification, and (2) investigating how an LLM makes use of the definitions. We thoroughly analyze the performance of GPT-3 model for fine-grained multi-label conspiracy theory classification of tweets using zero-shot labeling. In doing so, we asses how to improve the labeling by providing minimal but meaningful context in the form of the definitions of the labels. We compare descriptive noun phrases, humancrafted definitions, introduce a new method to help the model generate definitions from examples, and propose a method to evaluate GPT-3's understanding of the definitions. We demonstrate that improving definitions of class labels has a direct consequence on the downstream classification results.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 2 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions 引用了 Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
- ZeroGen: Efficient Zero-shot Learning via Dataset Generation 引用了 Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding 引用了 ZeroGen: Efficient Zero-shot Learning via Dataset Generation
- ZeroGen: Efficient Zero-shot Learning via Dataset Generation 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver? 引用了 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
引用它的顶会 Paper2
- Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document RevisionsQian Ruan, Ilia Kuznetsov, Iryna GurevychEMNLP 2024 · 2 次引用
- Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label DefinitionsSeyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff 等EMNLP 2025
它引用的 Paper9
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma 等NeurIPS 2022 · 22,562 次引用
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo 等NeurIPS 2022 · 8,168 次引用
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver?Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen 等EMNLP 2023 · 449 次引用
- Generating Training Data with Language Models: Towards Zero-Shot Language UnderstandingYu Meng, Jiaxin Huang, Yu Zhang, Jiawei HanNeurIPS 2022 · 309 次引用
- ZeroGen: Efficient Zero-shot Learning via Dataset GenerationJiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu 等EMNLP 2022 · 96 次引用
相关 Paper
- Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial DomainsMarcio Fonseca, Shay B. CohenACL Findings 2024 · 2 次引用
- Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt CollectionsRuiqi Zhong, Kristy Lee, Zheng Zhang, Dan KleinEMNLP Findings 2021 · 101 次引用
- The Benefits of Label-Description Training for Zero-Shot Text ClassificationLingyu Gao, Debanjan Ghosh, Kevin GimpelEMNLP 2023 · 6 次引用
- Improving Event Definition Following For Zero-Shot Event DetectionZefan Cai, Po-Nien Kung, Ashima Suvarna, Mingyu Derek Ma 等ACL 2024 · 3 次引用
- Coarse2Fine: Fine-grained Text Classification on Coarsely-grained Annotated DataDheeraj Mekala, Varun Gangal, Jingbo ShangEMNLP 2021 · 20 次引用
