Epipolar Consistency-based Network for Structure-Aware LF Semantic Segmentation

Light Field (LF) semantic segmentation relies on leveraging redundant information across multiple views to assign a semantic label to each pixel of the central view. Recent approaches typically feed the views into a pre-trained backbone and utilize an estimated depth map to aggregate semantic representations for label prediction. However, these methods ignore the correlation between encoded structural cues in LF and semantic labels. On one hand, it is challenging to identify matching points for regions that are occluded in some views. This broken view consistency emphasizes object edge localization, facilitating more precise edge labeling. On the other hand, the depth continuity for the same object ensures semantic consistency in adjacent regions. Therefore, effectively extracting structural cues and integrating them into semantic segmentation are key points in LF semantic segmentation.In this paper, we propose an Epipolar Consistency-based network for structure-aware LF semantic segmentation, termed ECNet. First, we explore the epipolar consistency between views to characterize the edges and depth cues of the input. Based on the embedded edges information, we design an edge-semantic correlation transformer to generate fine-grained representations of object edges. Furthermore, the proposed depth-semantic correlation transformer maps semantic features of one object closer together according the depth information.Extensive experiments demonstrate that ECNet achieves state-of-the-art performance, which reduces computational cost by 33.3% (in terms of FLOPs) while maintaining high segmentation accuracy.