Lune

AAAI2026Top-tier venue

Synergizing Multigrid Algorithms with Vision Transformer: A Novel Approach to Enhance the Seismic Foundation Model

Huiwen Wu, Shuo Zhang, Yi Liu, Hongbin Ye

Abstract

Due to the rapid advancement and homogenization of Artificial Intelligence (AI) technology development, transformer-based foundation models have revolutionized scientific applications, such as drug discovery, materials research, and astronomy. However, seismic data presents unique characteristics that require specialized processing techniques for pretraining foundation models in seismic contexts with high- and low-frequency features playing crucial roles. Existing Vision Transformer (ViT) with sequential image tokenization fails to efficiently and effectively capture both high- and low-frequency seismic information because they ignore the intrinsic structural patterns of seismograms. This work introduces ADATG, a novel adaptive two-grid training strategy with Hilbert encoding, explicitly tailored for seismogram data and leveraging the hierarchical structures inherent in seismic data. Specifically, our approach employs spectrum decomposition to separate high- and low-frequency components, and hierarchical Hilbert encoding to represent the data effectively. Moreover, inspired by the frequency principle, we propose an adaptive training strategy that initially emphasizes coarse-level information and then progressively refines the model's focus on fine-level features. Extensive experiments demonstrate the effectiveness and efficiency of our method. This research highlights the importance of data encoding and training strategies informed by the distinct characteristics of high- and low-frequency features in seismic images, ultimately enhancing the pretraining of visual seismic foundation models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers cites Masked Autoencoders Are Scalable Vision Learners
  • Masked Autoencoders Are Scalable Vision Learners cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on3

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision TransformersJihao Liu, Xin Huang, Jinliang Zheng, Yu Liu et al.CVPR 2023
  3. Masked Autoencoders Are Scalable Vision LearnersKaiming He, Xinlei Chen, Saining Xie, Yanghao Li et al.CVPR 2022

Related papers

  1. Adaptive Physics Transformer with Fused Global-Local Attention for Subsurface Energy SystemsXin Ju, Hadrian Fung, Yuyan Zhang, Carl Jacquemyn et al.ICML 2026 · 2 citations
  2. EEGiT: Teaching Vision Transformers to Understand the EEG signalJiahao Zhou, Chenghao Xu, Wei Wang, Erkun Yang et al.CVPR 2026
  3. Frequency-Dynamic Attention Modulation for Dense PredictionLinwei Chen, Lin Gu, Ying FuICCV 2025 · 13 citations
  4. Deep Compression of Pre-trained Transformer ModelsNaigang Wang, Chi-Chun (Charlie) Liu, Swagath Venkataramani, Sanchari Sen et al.NeurIPS 2022 · 38 citations
  5. Hierarchical Frequency-Decomposition Graph Neural Networks for Road Network Representation LearningJingtian Ma, Jingyuan Wang, Leong Hou UAAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines