Omni-Weather: A Unified Multimodal Model for Weather Radar Understanding and Generation
Zhiwang Zhou, Yuandong Pu, Xuming He, Yidi Liu, Yixin Chen, Junchao Gong, Xiang Zhuang, Wanghan Xu, Qinglong Cao, Shixiang Tang, Yihao Liu, Wenlong Zhang, Lei Bai
Abstract
Weather modeling requires both accurate prediction and mechanistic interpretation, yet existing methods treat these goals in isolation, separating generation from understanding. To address this gap, we present Omni-Weather, the first multimodal foundation model that unifies weather generation and understanding within a single architecture. Omni-Weather integrates a radar encoder for weather generation tasks, followed by unified processing using a shared self-attention mechanism. Moreover, we construct a Chain-of-Thought dataset for causal reasoning in weather generation, enabling interpretable outputs and improved perceptual quality. Extensive experiments show Omni-Weather achieves state-of-the-art performance in both weather generation and understanding. Our findings further indicate that generative and understanding tasks in the weather domain can mutually enhance each other. Omni-Weather also demonstrates the feasibility and value of unifying weather generation and understanding.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 7b63d6e8-07fd-445c-af24-7d4f524e1de6Builds on16
- ClimaX: A foundation model for weather and climateTung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K. Gupta et al.ICML 2023 · 426 citations
- Earthformer: Exploring Space-Time Transformers for Earth System ForecastingZhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu et al.NeurIPS 2022 · 410 citations
- SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite MeteorologyMark S. Veillette, Siddharth Samsi, Christopher J. MattioliNeurIPS 2020 · 179 citations
- PreDiff: Precipitation Nowcasting with Latent Diffusion ModelsZhihan Gao, Xingjian Shi, Boran Han, Hao Wang et al.NeurIPS 2023 · 171 citations
- LMFusion: Adapting Pretrained Language Models for Multimodal GenerationWeijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang et al.NeurIPS 2025 · 134 citations
Related papers
- WeatherGFM: Learning a Weather Generalist Foundation Model via In-context LearningXiangyu Zhao, Zhiwang Zhou, Wenlong Zhang, Yihao Liu et al.ICLR 2025
- RadarQA: Multi-modal Quality Analysis of Weather Radar ForecastsXuming He, Zhiyuan You, Junchao Gong, Couhua Liu et al.NeurIPS 2025 · 14 citations
- OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time ScalesTung Nguyen, Tuan Pham, Troy Arcomano, Rao Kotamarthi et al.NeurIPS 2025 · 12 citations
- VILA-U: a Unified Foundation Model Integrating Visual Understanding and GenerationYecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang et al.ICLR 2025
- Show-o: One Single Transformer to Unify Multimodal Understanding and GenerationJinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang et al.ICLR 2025
