EMNLP Findings2023
TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex Tasks
Shubhra Kanti Karmaker Santu, Dongji Feng
Abstract
While LLMs have shown great success in understanding and generating text in traditional conversational settings, their potential for performing ill-defined complex tasks is largely under-studied and yet to be benchmarked. However, conducting such benchmarking studies is challenging because of the large variations in LLMs' performance when different prompt types/styles are used and different degrees of detail are provided in the prompts. To address this issue, this paper proposes a general taxonomy that can be used to design prompts with specific properties in order to perform a wide range of complex tasks. This taxonomy will allow future benchmarking studies to report the specific categories of prompts used as part of the study, enabling meaningful comparisons across different studies. Also, by establishing a common standard through this taxonomy, researchers will be able to draw more accurate conclusions about LLMs' performance on a specific complex task.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Exploring Modular Prompt Design for Emotion and Mental Health Recognition cites Language Models are Few-Shot Learners
- Exploring Modular Prompt Design for Emotion and Mental Health Recognition cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Training language models to follow instructions with human feedback
- The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models cites Language Models are Few-Shot Learners
- The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models cites Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Argumentative Large Language Models for Explainable and Contestable Claim Verification cites Language Models are Few-Shot Learners
- Argumentative Large Language Models for Explainable and Contestable Claim Verification cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers12
- The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language ModelsJunyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng et al.ACL 2024 · 49 citations
- Argumentative Large Language Models for Explainable and Contestable Claim VerificationGabriel Freedman, Adam Dejl, Deniz Gorur, Xiang Yin et al.AAAI 2025 · 31 citations
- Using AI Assistants in Software Development: A Qualitative Study on Security Practices and ConcernsJan H. Klemmer, Stefan Albert Horstmann, Nikhil Patnaik, Cordelia Ludden et al.CCS 2024 · 14 citations
- Exploring Modular Prompt Design for Emotion and Mental Health RecognitionMinseo Kim, Taemin Kim, Thu Hoang Anh Vo, Yugyeong Jung et al.CHI 2025 · 11 citations
- UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them AllYuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, Lin WangCVPR 2024 · 8 citations
Builds on19
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
Related papers
- Are Large Language Models a Good Replacement of Taxonomies?Yushi Sun, Xin Hao, Kai Sun, Yifan Xu et al.VLDB 2024 · 26 citations
- Can Large Language Models Understand Real-World Complex Instructions?Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen et al.AAAI 2024 · 99 citations
- CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language ModelsTong Zhang, Peixin Qin, Yang Deng, Chen Huang et al.ACL 2024
- Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research PapersZhijian Xu, Yilun Zhao, Manasi Patwardhan, Lovekesh Vig et al.ACL 2025 · 17 citations
- AmbigNLG: Addressing Task Ambiguity in Instruction for NLGAyana Niwa, Hayate IsoEMNLP 2024 · 2 citations
