UniLLM: A Unified Large Language Model for Multi?Modal Urban Dynamics Prediction

Modern cities generate vast streams of urban dynamics data reflecting mobility demand, environmental conditions, and traffic patterns. The value of these data lies not only in individual modalities but in their integration—urban signals are highly interdependent, with changes in one modality often influencing others. Consequently, predicting any single urban dynamic requires information from multiple interrelated sources. Although numerous methods—ranging from deep learning models to recent LLM-based approaches—have been proposed, most are limited in scope. They either focus on single-modality prediction, rely on rigid model designs that lack flexibility, or overlook inter-modal dependencies. As a result, they struggle to adapt to dynamic urban conditions and suffer from degraded predictive performance across modalities. In this paper, we propose UniLLM, a unified large language model for multi-modal urban dynamics prediction. At its core, UniLLM introduces a Unified Cross-Modal Alignment Module that transforms heterogeneous urban data into latent representations while preserving modality-specific patterns and capturing cross-modal correlations through a contrastive learning objective. To support dynamic adaptation across tasks and modalities, we design a Routing-Aware Prompting Mechanism that learns soft prompts based on task context and modality semantics. Furthermore, a Multi-Modal Memory-Guided Adaptive Algorithm employs replay-based gradient coordination and Frank–Wolfe optimization to mitigate cross-modal catastrophic forgetting during fine-tuning. Extensive experiments across multiple cities and urban modalities demonstrate that UniLLM consistently outperforms state-of-the-art baselines. These results highlight UniLLM's potential as a flexible and robust forecasting model for real-world, multi-modal urban environments.