UniSumEval: Towards Unified, Fine-grained, Multi-dimensional Summarization Evaluation for LLMs

Existing benchmarks for summarization quality evaluation often lack diverse input scenarios, focus on narrowly defined dimensions (e.g., faithfulness), and struggle with subjective and coarse-grained annotation schemes. To address these shortcomings, we create UNISUMEVAL benchmark, which extends the range of input context (e.g., domain, length) and provides finegrained, multi-dimensional annotations. We use AI assistance in data creation, identifying potentially hallucinogenic input texts, and also helping human annotators reduce the difficulty of fine-grained annotation tasks.With UNISUMEVAL, we benchmark nine latest language models as summarizers, offering insights into their performance across varying input contexts and evaluation dimensions. Furthermore, we conduct a thorough comparison of SOTA automated summary evaluators. Our benchmark data will be available at https:// github.com/DISL-Lab/UniSumEval-v1.0.