Lune

CVPR2024Top-tier venue

General Point Model Pretraining with Autoencoding and Autoregressive

Zhe Li, Zhangyang Gao, Cheng Tan, Bocheng Ren, Laurence T. Yang, Stan Z. Li

Abstract

The pre-training architectures of large language models encompass various types, including autoencoding models, autoregressive models, and encoder-decoder models. We posit that any modality can potentially benefit from a large language model, as long as it undergoes vector quantization to become discrete tokens. Inspired by the General Language Model, we propose a General Point Model (GPM) that seamlessly integrates autoencoding and autoregressive tasks in a point cloud transformer. This model is versatile, allowing fine-tuning for downstream point cloud representation tasks, as well as unconditional and conditional generation tasks. GPM enhances masked prediction in autoencoding through various forms of mask padding tasks, leading to improved performance in point cloud understanding. Additionally, GPM demonstrates highly competitive results in unconditional point cloud generation tasks, even exhibiting the potential for conditional generation tasks by modifying the input's conditional information. Compared to models like Point-BERT, MaskPoint, and PointMAE, our GPM achieves superior performance in point cloud understanding tasks. Furthermore, the integration of autoregressive and autoencoding within the same transformer underscores its versatility across different downstream tasks. Codes are available at https://github.com/gentlefress/GPM

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BEiT: BERT Pre-Training of Image Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • What We Miss Matters: Learning from the Overlooked in Point Cloud Transformers cites BEiT: BERT Pre-Training of Image Transformers
  • Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. PointMC: Multi-view Consistent Encoding and Center-Global Feature Fusion for Point Clouds UnderstandingXinxing Yu, Ajian Liu, Sunyuan Qiang, Yuzhong Wang et al.AAAI 2026 · 1 citation
  2. What We Miss Matters: Learning from the Overlooked in Point Cloud TransformersYi Wang, Jiaze Wang, Ziyu Guo, Renrui Zhang et al.NeurIPS 2025 · 1 citation

Builds on27

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
  3. Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
  4. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
  5. BEiT: BERT Pre-Training of Image TransformersHangbo Bao, Li Dong, Songhao Piao, Furu WeiICLR 2022 · 3,632 citations

Related papers

  1. PointGPT: Auto-regressively Generative Pre-training from Point CloudsGuangyan Chen, Meiling Wang, Yi Yang, Kai Yu et al.NeurIPS 2023 · 219 citations
  2. Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point ModelingXumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang et al.CVPR 2022 · 684 citations
  3. GLM: General Language Model Pretraining with Autoregressive Blank InfillingZhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding et al.ACL 2022
  4. GPT4Point: A Unified Framework for Point-Language Understanding and GenerationZhangyang Qi, Ye Fang, Zeyi Sun, Xiaoyang Wu et al.CVPR 2024 · 23 citations
  5. Point Cloud as a Foreign Language for Multi-modal Large Language ModelSneha Paul, Zachary Patterson, Nizar BouguilaCVPR 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines