CES: Combinatorial Experts Selection via Contextual Linear Bandits

With the rapid advancement of large language models (LLMs), multi-agent systems have emerged as a promising alternative to scaling up a single model. Existing approaches ensemble multiple LLMs to improve response quality, but they often rely on static prior knowledge of model capabilities and prompts, and require extensive parameter tuning. Some of the methods also treat each combination of LLMs as a learning objective, which leads to exponential time complexity. In this work, we propose an offline-to-online combinatorial experts selection (CES) framework to address these limitations. CES leverages offline evaluation to warm-start model capability estimation and employs online learning to adapt to capability shifts and correct offline inaccuracies. By integrating model features and input semantic representations into a combinatorial multi-armed bandit formulation, CES captures the interaction between prompts and LLMs without introducing complex auxiliary structures such as knowledge graphs. Modeling each LLM as a base arm with answer quality represented by a linear function of model and prompt features, CES achieves polynomial time complexity while excellently balancing performance and cost. Our experiments, conducted on popular LLM evaluation datasets such as AlpacaEval 2.0, show CES's effectiveness, laying the groundwork for future extensions.