Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-training
Renrui Zhang, Ziyu Guo, Peng Gao, Rongyao Fang, Bin Zhao, Dong Wang, Yu Qiao, Hongsheng Li
Abstract
Masked Autoencoders (MAE) have shown great potentials in self-supervised pretraining for language and 2D image transformers. However, it still remains an open question on how to exploit masked autoencoding for learning 3D representations of irregular point clouds. In this paper, we propose Point-M2AE, a strong Multi-scale MAE pre-training framework for hierarchical self-supervised learning of 3D point clouds. Unlike the standard transformer in MAE, we modify the encoder and decoder into pyramid architectures to progressively model spatial geometries and capture both fine-grained and high-level semantics of 3D shapes. For the encoder that downsamples point tokens by stages, we design a multi-scale masking strategy to generate consistent visible regions across scales, and adopt a local spatial self-attention mechanism during fine-tuning to focus on neighboring patterns. By multi-scale token propagation, the lightweight decoder gradually upsamples point tokens with complementary skip connections from the encoder, which further promotes the reconstruction from a global-to-local perspective. Extensive experiments demonstrate the state-of-the-art performance of Point-M2AE for 3D representation learning. With a frozen encoder after pretraining, Point-M2AE achieves 92.9% accuracy for linear SVM on ModelNet40, even surpassing some fully trained methods. By fine-tuning on downstream tasks, Point-M2AE achieves 86.43% accuracy on ScanObjectNN, +3.36% to the secondbest, and largely benefits the few-shot classification, part segmentation and 3D object detection with the hierarchical pre-training scheme. Code is available at https://github.com/ZrrSkywalker/Point-M2AE .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
124 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- PointGPT: Auto-regressively Generative Pre-training from Point Clouds cites A Simple Framework for Contrastive Learning of Visual Representations
- PointGPT: Auto-regressively Generative Pre-training from Point Clouds cites Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining
- PointGPT: Auto-regressively Generative Pre-training from Point Clouds cites Language Models are Few-Shot Learners
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites A Simple Framework for Contrastive Learning of Visual Representations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning cites Language Models are Few-Shot Learners
- PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning cites Learning Transferable Visual Models From Natural Language Supervision
- Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining cites A Simple Framework for Contrastive Learning of Visual Representations
- Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining cites Language Models are Few-Shot Learners
- Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining cites Learning Transferable Visual Models From Natural Language Supervision
- PointMamba: A Simple State Space Model for Point Cloud Analysis cites PointGPT: Auto-regressively Generative Pre-training from Point Clouds
- PointMamba: A Simple State Space Model for Point Cloud Analysis cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- PointMamba: A Simple State Space Model for Point Cloud Analysis cites Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining
- PointMamba: A Simple State Space Model for Point Cloud Analysis cites Language Models are Few-Shot Learners
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites A Simple Framework for Contrastive Learning of Visual Representations
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
- Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers124
- PointMamba: A Simple State Space Model for Point Cloud AnalysisDingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu et al.NeurIPS 2024 · 380 citations
- PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world LearningXiangyang Zhu, Renrui Zhang, Bowei He, Ziyu Guo et al.ICCV 2023 · 248 citations
- Chat-Scene: Bridging 3D Scene and Large Language Models with Object IdentifiersHaifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang et al.NeurIPS 2024 · 230 citations
- PointGPT: Auto-regressively Generative Pre-training from Point CloudsGuangyan Chen, Meiling Wang, Yi Yang, Kai Yu et al.NeurIPS 2023 · 219 citations
- Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative PretrainingZekun Qi, Runpei Dong, Guofan Fan, Zheng Ge et al.ICML 2023 · 209 citations
Builds on29
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
Related papers
- Learning 3D Representations from 2D Pre-Trained Models via Image-to-Point Masked AutoencodersRenrui Zhang, Liuhui Wang, Yu Qiao, Peng Gao et al.CVPR 2023
- Regress Before Construct: Regress Autoencoder for Point Cloud Self-supervised LearningYang Liu, Chen Chen, Can Wang, Xulin King et al.ACM MM 2023 · 13 citations
- Point Cloud Self-Supervised Learning via 3D to Multi-View Masked LearnerZhimin Chen, Xuewei Chen, Xiao Guo, Yingwei Li et al.ICCV 2025 · 1 citation
- Point-MaDi: Masked Autoencoding with Diffusion for Point Cloud Pre-trainingXiaoyang Xiao, Runzhao Yao, Zhiqiang Tian, Shaoyi DuNeurIPS 2025 · 4 citations
- PCP-MAE: Learning to Predict Centers for Point Masked AutoencodersXiangdong Zhang, Shaofeng Zhang, Junchi YanNeurIPS 2024 · 44 citations
