BuildingGPT: Auto-Regressive Building Wireframe Reconstruction Model with Reinforcement Learning

In this paper, we propose BuildingGPT, a novel autoregressive model for building wireframe reconstruction from point clouds with reinforcement learning. Unlike prior works based on detection or diffusion models, Build-ingGPT reformulates the building wireframe reconstruction task into a sequence prediction problem. Based on the hierarchical building wireframe tokenization, the wireframe sequences are organized in a structurally-and semantically-aware order for the next-token prediction. The point cloud encoder first transforms the input point cloud into a fixed-length latent code prepended before the wireframe sequence. Then, BuildingGPT auto-regressively predicts tokens conditioned on the latent code. After detokenization, the building wireframe is obtained. To enhance the model performance, we adopt a two-stage training paradigm including the pre-training and post-training. After the auto-regressive pre-training, Direct Preference Optimization (DPO) is employed as a post-training strategy to align reconstruction results with human preferences. Extensive experiments on the large-scale MunichWF dataset show that BuildingGPT outperforms existing state-of-theart methods. Our code is available at: https : / / github.com/3dv-casia/BuildingGPT/.