ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities

Traditional fixed test datasets fall short in evaluating the open-ended capabilities of foundation models. To address this, we propose ONEBench (OpeN-Ended Benchmarking), a new paradigm that consolidates individual evaluation datasets into a unified, ever-expanding sample pool. ONEBench enables custom benchmarks for specific capabilities while reusing and aggregating samples, mitigating overfitting and dataset bias for broader capability assessment. It reframes model evaluation as selecting and aggregating samplelevel tests. Transitioning from task-specific benchmarks to ONEBench introduces two challenges: heterogeneity (aggregating diverse metrics) and incompleteness (comparing models tested on different data subsets). To address these, we propose an aggregation algorithm that ensures identifiability-asymptotically recovering ground-truth scores-and rapid convergence, enabling accurate model comparisons with relatively little data. On homogenous datasets, our algorithm produces rankings that highly correlate with average scores. Moreover, it remains robust to over 95% missing measurements, reducing evaluation costs by up to 20 times. We introduce ONEBench-LLM for language models and ONEBench-LMM for visionlanguage models, enabling targeted model testing across diverse capabilities. * Equal contribution, random order, • core contributors 1 From a talk by Alexei Efros at ICML 2020