EMNLP Findings2023
Definitions Matter: Guiding GPT for Multi-label Classification
Youri Peskine, Damir Korencic, Ivan Grubisic, Paolo Papotti, Raphaël Troncy, Paolo Rosso
Abstract
Large language models have recently risen in popularity due to their ability to perform many natural language tasks without requiring any fine-tuning. In this work, we focus on two novel ideas: (1) generating definitions from examples and using them for zero-shot classification, and (2) investigating how an LLM makes use of the definitions. We thoroughly analyze the performance of GPT-3 model for fine-grained multi-label conspiracy theory classification of tweets using zero-shot labeling. In doing so, we asses how to improve the labeling by providing minimal but meaningful context in the form of the definitions of the labels. We compare descriptive noun phrases, humancrafted definitions, introduce a new method to help the model generate definitions from examples, and propose a method to evaluate GPT-3's understanding of the definitions. We demonstrate that improving definitions of class labels has a direct consequence on the downstream classification results.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions cites Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
- ZeroGen: Efficient Zero-shot Learning via Dataset Generation cites Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
- Generating Training Data with Language Models: Towards Zero-Shot Language Understanding cites ZeroGen: Efficient Zero-shot Learning via Dataset Generation
- ZeroGen: Efficient Zero-shot Learning via Dataset Generation cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver? cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers2
- Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document RevisionsQian Ruan, Ilia Kuznetsov, Iryna GurevychEMNLP 2024 · 2 citations
- Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label DefinitionsSeyedali Mohammadi, Bhaskara Hanuma Vedula, Hemank Lamba, Edward Raff et al.EMNLP 2025
Builds on9
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver?Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen et al.EMNLP 2023 · 449 citations
- Generating Training Data with Language Models: Towards Zero-Shot Language UnderstandingYu Meng, Jiaxin Huang, Yu Zhang, Jiawei HanNeurIPS 2022 · 309 citations
- ZeroGen: Efficient Zero-shot Learning via Dataset GenerationJiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu et al.EMNLP 2022 · 96 citations
Related papers
- Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial DomainsMarcio Fonseca, Shay B. CohenACL Findings 2024 · 2 citations
- Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt CollectionsRuiqi Zhong, Kristy Lee, Zheng Zhang, Dan KleinEMNLP Findings 2021 · 101 citations
- The Benefits of Label-Description Training for Zero-Shot Text ClassificationLingyu Gao, Debanjan Ghosh, Kevin GimpelEMNLP 2023 · 6 citations
- Improving Event Definition Following For Zero-Shot Event DetectionZefan Cai, Po-Nien Kung, Ashima Suvarna, Mingyu Derek Ma et al.ACL 2024 · 3 citations
- Coarse2Fine: Fine-grained Text Classification on Coarsely-grained Annotated DataDheeraj Mekala, Varun Gangal, Jingbo ShangEMNLP 2021 · 20 citations
