Cross-Modal Coreference Alignment: Enabling Reliable Information Transfer in Omni-LLMs

Omni Large Language Models (Omni-LLMs) have demonstrated impressive capabilities in holistic multi-modal perception, yet they consistently falter in complex scenarios requiring synergistic omni-modal reasoning. Beyond understanding global multimodal context, effective reasoning also hinges on fine-grained crossmodal alignment, especially identifying shared referents across modalities, yet this aspect has been largely overlooked. To bridge this gap, we formalize the challenge as a cross-modal coreference problem, where a model must localize a referent in a source modality and reidentify it in a target modality. Building on this paradigm, we introduce CROSSOMNI, a dataset comprising nine tasks equipped with human-designed reasoning rationales to evaluate and enhance this capability. Experiments on 13 Omni-LLMs reveal systematic weaknesses in cross-modal coreference, which we attribute to the absence of coreference-aware thinking patterns. To address this, we enhance crossmodal alignment via two strategies: a trainingfree In-Context Learning method and a trainingbased SFT+GRPO framework designed to induce such thinking patterns. Both approaches yield substantial performance gains and generalize effectively to collaborative reasoning tasks. Overall, our findings highlight crossmodal coreference as a crucial missing piece for advancing robust omni-modal reasoning. Auto-Generation Human Verification Double Check Consistency Test-set Distribution Q: What is the occupation of the person who said ''when did he get out of prison''? Option: A. nurse B. officer C. sergeant D. detective