"A good pun is its own reword": Can Large Language Models Understand Puns?

As one of the common rhetorical devices, puns play a vital role in linguistic study, including the comprehensive analysis of linguistic humor. Although large language models (LLMs) have been widely explored on various tasks of natural language understanding and generation, their ability to understand puns has not been systematically studied, limiting the utilization of LLMs in creative writing and humor creation. In this paper, we leverage three popular tasks, i.e., pun recognition, pun explanation, and pun generation, to systematically evaluate LLMs' capability of understanding puns. In addition to the evaluation metrics adopted by prior research, we introduce some new evaluation methods and metrics that are better suited to the in-context learning paradigm of LLMs. These new metrics offer a more rigorous assessment of an LLM's capability to understand puns and align more closely with human cognition. Our research findings reveal the "lazy pun generation" pattern and identify the primary challenges in understanding puns with LLMs. The resources of this paper will be released upon publication.