Lune

EMNLP Findings2023

The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation

Tyler Loakman, Aaron Maladry, Chenghua Lin

Abstract

Human evaluation is often considered to be the gold standard method of evaluating a Natural Language Generation system. However, whilst its importance is accepted by the community at large, the quality of its execution is often brought into question. In this position paper, we argue that the generation of more esoteric forms of language -humour, irony and sarcasm -constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance, and every effort should be made to report demographic characteristics wherever possible, in the interest of transparency and replicability. We support these claims with an overview of each language form and an analysis of examples in terms of how their interpretation is affected by different participant variables. We additionally perform a critical survey of recent works in NLG to assess how well evaluation procedures are reported in this subdomain, and note a severe lack of open reporting of evaluator demographic information, and a significant reliance on crowdsourcing platforms for recruitment.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Cited by top-tier papers3

  1. MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical LanguageShun Wang, Ge Zhang, Han Wu, Tyler Loakman et al.EMNLP 2024 · 3 citations
  2. In Benchmarks We Trust ... Or Not?Ine Gevers, Victor De Marez, Jens Van Nooten, Jens Lemmens et al.EMNLP 2025 · 1 citation
  3. Drivel-ology: Challenging LLMs with Interpreting Nonsense with DepthYang Wang, Chenghao Xiao, Chia-Yi Hsiao, Zi Yan Chang et al.EMNLP 2025

Builds on13

  1. KGPT: Knowledge-Grounded Pre-Training for Data-to-Text GenerationWenhu Chen, Yu Su, Xifeng Yan, William Yang WangEMNLP 2020 · 115 citations
  2. Content Planning for Neural Story Generation with Aristotelian RescoringSeraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph M. Weischedel, Nanyun PengEMNLP 2020 · 106 citations
  3. Iteratively Prompt Pre-trained Language Models for Chain of ThoughtBoshi Wang, Xiang Deng, Huan SunEMNLP 2022 · 62 citations
  4. R^3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense KnowledgeTuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, Nanyun PengACL 2020 · 58 citations
  5. Automatic Poetry Generation from Prosaic TextTim Van de CruysACL 2020 · 54 citations

Related papers

  1. "A good pun is its own reword": Can Large Language Models Understand Puns?Zhijun Xu, Siyu Yuan, Lingjie Chen, Deqing YangEMNLP 2024 · 6 citations
  2. Can Language Models Make Fun? A Case Study in Chinese Comical CrosstalkJianquan Li, Xiangbo Wu, Xiaokang Liu, Qianqian Xie et al.ACL 2023 · 2 citations
  3. Assessing the Capabilities of LLMs in Humor: A Multi-dimensional Analysis of Oogiri Generation and EvaluationRitsu Sakabe, Hwichan Kim, Tosho Hirasawa, Mamoru KomachiAAAI 2026
  4. ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language ModelsAparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati et al.ACL 2024 · 19 citations
  5. Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text GenerationKatelyn Xiaoying Mei, Yi-Li Hsu, Minjoon Choi, Zongwan Cao et al.ACL 2026
Dusk over the sea between two cliffs drawn in fine vertical lines