Cross-view Transformers for real-time Map-view Semantic Segmentation
Brady Zhou, Philipp Krähenbühl
Abstract
We present cross-view transformers, an efficient attention-based model for map-view semantic segmentation from multiple cameras. Our architecture implicitly learns a mapping from individual camera views into a canonical map-view representation using a camera-aware cross-view attention mechanism. Each camera uses positional embeddings that depend on its intrinsic and extrinsic calibration. These embeddings allow a transformer to learn the mapping across different views without ever explicitly modeling it geometrically. The architecture consists of a convolutional image encoder for each view and cross-view transformer layers to infer a map-view semantic segmentation. Our model is simple, easily parallelizable, and runs in realtime. The presented architecture performs at state-of-the-art on the nuScenes dataset, with 4x faster inference speeds. Code is available at https://github.com/bradyz/cross_view_transformers.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
92 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Scene as Occupancy cites nuScenes: A Multimodal Dataset for Autonomous Driving
- OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction cites nuScenes: A Multimodal Dataset for Autonomous Driving
- VectorMapNet: End-to-end Vectorized HD Map Learning cites nuScenes: A Multimodal Dataset for Autonomous Driving
- VAD: Vectorized Scene Representation for Efficient Autonomous Driving cites nuScenes: A Multimodal Dataset for Autonomous Driving
- VAD: Vectorized Scene Representation for Efficient Autonomous Driving cites VectorMapNet: End-to-end Vectorized HD Map Learning
- PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images cites nuScenes: A Multimodal Dataset for Autonomous Driving
- PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images cites 3D-LaneNet: End-to-End 3D Multiple Lane Detection
Cited by top-tier papers92
- VAD: Vectorized Scene Representation for Efficient Autonomous DrivingBo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao et al.ICCV 2023 · 602 citations
- PETRv2: A Unified Framework for 3D Perception from Multi-Camera ImagesYingfei Liu, Junjie Yan, Fan Jia, Shuailin Li et al.ICCV 2023 · 513 citations
- OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy PredictionYunpeng Zhang, Zheng Zhu, Dalong DuICCV 2023 · 354 citations
- VectorMapNet: End-to-end Vectorized HD Map LearningYicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang et al.ICML 2023 · 332 citations
- Scene as OccupancyWenwen Tong, Chonghao Sima, Tai Wang, Li Chen et al.ICCV 2023 · 251 citations
Builds on8
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional DomainsMatthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil et al.NeurIPS 2020 · 4,036 citations
- Digging Into Self-Supervised Monocular Depth EstimationClément Godard, Oisin Mac Aodha, Michael Firman, Gabriel J. BrostowICCV 2019 · 2,416 citations
- Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous DrivingXinzhu Ma, Zhihui Wang, Haojie Li, Pengbo Zhang et al.ICCV 2019 · 339 citations
- 3D-LaneNet: End-to-End 3D Multiple Lane DetectionNoa Garnett, Rafi Cohen, Tomer Pe'er, Roee Lahav et al.ICCV 2019 · 232 citations
- nuScenes: A Multimodal Dataset for Autonomous DrivingHolger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora et al.CVPR 2020
Related papers
- Object Pose Transformer: Unifying Unseen Object Pose EstimationWeihang Li, Lorenzo Garattoni, Fabien Despinoy, Nassir Navab, Benjamin BusamCVPR Findings 2026
- Learning Multi-Scene Absolute Pose Regression with TransformersYoli Shavit, Ron Ferens, Yosi KellerICCV 2021 · 163 citations
- BAEFormer: Bi-Directional and Early Interaction Transformers for Bird's Eye View Semantic SegmentationCong Pan, Yonghao He, Junran Peng, Qian Zhang et al.CVPR 2023
- UniTR: A Unified and Efficient Multi-Modal Transformer for Bird's-Eye-View RepresentationHaiyang Wang, Hao Tang, Shaoshuai Shi, Aoxue Li et al.ICCV 2023 · 106 citations
- DVGT: Driving Visual Geometry TransformerSicheng Zuo, Zixun Xie, Wenzhao Zheng, Shaoqing Xu et al.CVPR 2026 · 23 citations
