NVILA: Efficient Frontier Visual Language Models

Decoding Speed (token/s) (1.9-5.1X Faster) (Pre-fill: 1.6-2.2X Faster / Decode: 1.2-2.8X Faster) 5.1X Faster 2.2X Faster 2.8X Faster AI2D C h a rt Q A D oc VQ A In fo VQ A M a th V is ta MMMU R e a lW o rl d Q A S EE D (im ag e) Te xt VQ A V Q A v 2 (c) Accuracy on image and video benchmarks (On-par or superior accuracy on all benchmarks)

Efficient Frontier VLMs. (a) NVILA trains image and video models 5.1→ and 1.9→ faster, respectively, than LLaVA-OneVision (OV), which is the only baseline model with publicly disclosed training costs. (b) Against Qwen2-VL, NVILA achieves a 1.6-2.2→ measured speedup in the pre-filling stage and a 1.2-2.8→ speedup during the decoding stage. (c) NVILA's efficiency is achieved without compromising accuracy; in fact, it delivers comparable or even superior accuracy across image and video benchmarks. All models in this table have 8B parameters. Training time in (a) is measured using NVIDIA H100 GPUs, while inference speed in (b) is measured using a single NVIDIA GeForce RTX 4090 GPU. Accuracy numbers in (c) are normalized relative to the highest score for each benchmark.