InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception

3D scene understanding is vital for applications in autonomous driving, robotics, and augmented reality. However, scene understanding based on 3D Gaussian Splatting faces three key challenges: (i) an imbalance between appearance and semantics, (ii) inconsistencies in object boundaries, and (iii) difficulties with top-down instance segmentation. To address these challenges, we propose InstanceGaussian, a method that jointly learns appearance and semantic features while adaptively aggregating instances. Our contributions are as follows: (i) a new Semantic-Scaffold-GS representation to improve feature representation and boundary delineation, (ii) a progressive training strategy for enhanced stability and segmentation, and (iii) a category-agnostic, bottom-up instance aggregation approach for better segmentation. Experimental results demonstrate that our approach achieves state-of-the-art performance in category-agnostic, openvocabulary 3D point-level segmentation, validating the effectiveness of our proposed method. Project page: https://lhj-git.github.io/InstanceGaussian/ * Corresponding author This work is supported by Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology(Grant No. 2024B1212010006) * The term "semantic" is used here to distinguish it from "appearance". In subsequent sections and code implementations, this concept will be explicitly referred to as "instance features".