Fico: Evaluating Vision-Language Models under Visual Fidelity and Compression at Scale

Visual text compression is an emerging paradigm for rendering text as images for processing by vision-language models (VLMs), enabling higher information density per context token. However, the robustness of VLMs under dense, text-based visual inputs remains unevaluated. We introduce F ICO , a benchmark designed to assess VLM robustness across seven controlled variants of visual fidelity and information density. F ICO spans documents of 8k to 64k tokens and includes three tasks of increasing semantic granularity: optical character recognition (OCR), needle-in-a-haystack (NIAH) retrieval, and visual question answering (VQA). Evaluating 13 general-purpose VLMs and 3 OCR-specialized models reveals three consistent trends: performance drops sharply under increased density or reduced resolution; cross-task transfer between OCR, NIAH, and VQA is limited; and VQA is comparatively robust, suggesting that low-level details are lost before semantics. By exposing failure modes that remain invisible under conventional VLM evaluations, F ICO establishes a rigorous test-bed for visual text compression. Data and code are available at https://github.com/ wang-research-lab/fico-bench .