GenColorBench: A Color Evaluation Benchmark for Text-to-Image Generation

Recent years have seen impressive advances in text-toimage generation, with image generative or unified models, generating high-quality images from text. Yet these models still struggle with fine-grained color control, often failing to accurately match colors specified in text prompts. While existing benchmarks evaluate compositional reasoning and prompt adherence, none systematically assess the color precision. Color is fundamental to human visual perception and communication, and critical for applications from art to design workflows requiring brand consistency. However, current benchmarks either neglect color or rely on coarse assessments, missing key capabilities like interpreting RGB values or aligning with human expectations. To this end, we propose GenColorBench, the first comprehensive benchmark for T2I color generation, grounded in color systems like ISCC-NBS and CSS3/X11, including numerical colors which are absent elsewhere. With 44K color-focused prompts covering 400+ colors, it reveals models' true capabilities via perceptual and automated assessments. Evaluations of popular T2I models on GenCol-orBench reveal significant performance variation, indicating which color conventions models understand the best and exposing their failure modes. Furthermore, GenCol-orBench provides insights to guide future improvements in precise color generation. The benchmark is available at https://moatifbutt.github.io/gencolorbench/.