ABPNet: Adaptive Background Modeling for Generalized Few Shot Segmentation

Existing Few Shot Segmentation (FS-Seg) methods mostly study a restricted setting where only foreground and background are required to be discriminated and fall short at discriminating multiple classes. In this paper, we focus on a challenging but more practical variant: Generalized Few Shot Segmentation (GFS-Seg), where all SEEN and UNSEEN classes are segmented simultaneously. Previous methods treat the background as a regular class, leading to difficulty in differentiating UNSEEN classes from it at the test stage. To address this issue, we propose Adaptive Background Modeling and Prototype Query Network (ABPNet), in which the background is formulated as the complement of the set of interested classes. With the help of the attention mechanism and a novel meta-training strategy, it learns an effective set difference function that predicts task-specific background adaptively. Furthermore, we design a Prototype Querying (PQ) module that effectively transfers the learned knowledge to UNSEEN classes with a neural dictionary. Experimental results demonstrate that ABPNet significantly outperforms the state-of-the-art method CAPL on PASCAL-5i and COCO-20i, especially on UNSEEN classes. Also, without retraining, ABPNet can generalize well to FS-Seg.