LMM4LMM: Benchmarking and Evaluating Large-Multimodal Image Generation With LMMs

2100 (a) Step 1: Prompt collection (b) Step 2: T2I Generation (c) Step 3: Human annotation 16 Annotators 50,400 Generated images <image> <prompt> "a blue cow" Perception (clarity, authenticity, aesthetics) T2I Correspondence (text-image alignment) Does the image contain a cow in the color blue? 0 5 0 5 Yes No 100,800 MOSs from 2 evaluation perspectives 50,400 Yes or no question answering pairs (d) Step 4: Model Design LMM4LMM <A photo of 4 boats> Image 24 LMM-T2I models 50K Images Generated from 2100 Prompts 100K MOSs Assessed from 2 Perspectives 50K Yes or No Question Answer Pairs on 20 Tasks (e) Step 5: Model Comparison LMM EvalMi-50k How would you rate the Perception quality of this image? How would you rate the Correspondence of this image and its prompt? <Prompt> Does the image contain 4 boats? Answer yes or no. A1: The Perception quality of the image is good. Score: 58.21 A2: The Correspondence of the image and its prompt is Bad. Score: 33.33 A3: No. The image does not contain 4 boats. 20 Fine-grained tasks Figure 1. We present the large multimodal image generation evaluation database and model, termed EvalMi-50K and LMM4LMM, respectively. (a) We first collect 2100 comprehensive prompts across 20 fine-grained tasks. (b) Then 24 LMM-T2I models are applied to generate 50K images. (c) 100K MOSs and 50K question-answering pairs are acquired from 16 annotators. (d) We design LMM4LMM to evaluate LMM-T2I models. (e) We conduct model comparisons on EvalMi-50K and the other 7 benchmarks.