ReFound: Crafting a Foundation Model for Urban Region Understanding upon Language and Visual Foundations

Understanding urban regional characteristics is pivotal in driving critical insights for urban planning and management. We have witnessed the successful application of pre-trained Foundation Models (FMs) in generating universal representations for various downstream tasks. However, applying this principle to the geospatial domain remains challenging, primarily due to the difficulty of gathering extensive data for developing a dedicated urban foundation model. Though there have been some attempts to empower the existing FMs with urban data, most of them focus on single-modality FMs without considering the multi-modality nature of urban region understanding tasks. To address this gap, we introduce ReFound - a novel framework for <u>Re</u>-training a <u>Found</u>ation model for urban region understanding, harnessing the strengths of both language and visual FMs. In this framework, we first invent a Mixture-of-Geospatial-Expert (MoGE) Transformer, to effectively integrate the embedding of multi-source geospatial data. Building on this, ReFound is enhanced by jointly distilling knowledge from language, visual, and visual-language FMs respectively, thus augmenting its generalization capabilities. Meanwhile, we design a masked geospatial data modeling approach alongside a cross-modal spatial alignment mechanism, to enhance the spatial knowledge of ReFound derived from geospatial data. Extensive experiments conducted on six real-world datasets over three urban region understanding tasks demonstrate the superior performance of our framework.