Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability

In generative commonsense reasoning tasks such as CommonGen, generative large language models (LLMs) compose sentences that include all given concepts. However, when focusing on instruction-following capabilities, if a prompt specifies a concept order, LLMs must generate sentences that adhere to the specified order. To address this, we propose Ordered CommonGen, a benchmark designed to evaluate the compositional generalization and instruction-following abilities of LLMs. This benchmark measures ordered coverage to assess whether concepts are generated in the specified order, enabling a simultaneous evaluation of both abilities. We conducted a comprehensive analysis using 36 LLMs and found that, while LLMs generally understand the intent of instructions, biases toward specific concept order patterns often lead to low-diversity outputs or identical results even when the concept order is altered. Moreover, even the most instructioncompliant LLM achieved only about 75% ordered coverage, highlighting the need for improvements in both instruction-following and compositional generalization capabilities. 3 https://github.com/neural-dialogue-metrics/ Distinct-N 4 https://hf.co/openai-community/gpt2-xl 5 https://github.com/asahi417/lmppl Concepts Coverage (↑) Similarlity (↓) Diversity (↑) Perplexity (↓) w/o order w/ order Ordered Rate pBLEU pBLEURT Distinct Diverse Rate