CAML: Collaborative Auxiliary Modality Learning for Multi-Agent Systems

Multi-modal learning has emerged as a key technique for improving performance across domains such as autonomous driving, robotics, and reasoning. However, in certain scenarios, particularly in resource-constrained environments, some modalities available during training may be absent during inference. While existing frameworks effectively utilize multiple data sources during training and enable inference with reduced modalities, they are primarily designed for single-agent settings. This poses a critical limitation in dynamic environments such as connected autonomous vehicles (CAV), where incomplete data coverage can lead to decisionmaking blind spots. Conversely, some works explore multi-agent collaboration but without addressing missing modality at test time. To overcome these limitations, we propose Collaborative Auxiliary Modality Learning (CAML), a novel multi-modal multi-agent framework that enables agents to collaborate and share multi-modal data during training, while allowing inference with reduced modalities during testing. Experimental results in collaborative decision-making for CAV in accident-prone scenarios demonstrate that CAML achieves up to a 58.1% improvement in accident detection. Additionally, we validate CAML on real-world aerial-ground robot data for collaborative semantic segmentation, achieving up to a 10.6% improvement in mIoU.

Recent work in machine learning [9,10,16,37,45] has sought to address these problems by enabling models to leverage additional modalities during training while supporting inference with fewer modalities. Such approaches reduce computational costs and improve robustness in conditions where some sensors may be unavailable at inference time. Shen et al. [41] formalized these tasks under the framework of Auxiliary Modality Learning (AML), which effectively reduces the dependency on expensive or unreliable modalities.

Despite the benefits of AML, notable limitations remain. AML is primarily designed for single-agent settings, where an individual model is trained to handle reduced modalities during inference. A key challenge in current AML frameworks is insufficient data coverage, particularly in dynamic environments such as connected autonomous vehicles (CAV). In these scenarios, a single agent's data 39th Conference on Neural Information Processing Systems (NeurIPS 2025).