OrienterNet: Visual Localization in 2D Public Maps with Neural Matching

Humans can orient themselves in their 3D environments using simple 2D maps. Differently, algorithms for visual localization mostly rely on complex 3D point clouds that are expensive to build, store, and maintain over time. We bridge this gap by introducing OrienterNet, the first deep neural network that can localize an image with sub-meter accuracy using the same 2D semantic maps that humans use. Orienter-Net estimates the location and orientation of a query image by matching a neural Bird's-Eye View with open and globally available maps from OpenStreetMap, enabling anyone to localize anywhere such maps are available. OrienterNet is supervised only by camera poses but learns to perform semantic matching with a wide range of map elements in an endto-end manner. To enable this, we introduce a large crowdsourced dataset of images captured across 12 cities from the diverse viewpoints of cars, bikes, and pedestrians. Orienter-Net generalizes to new datasets and pushes the state of the art in both robotics and AR scenarios. The code is available at github.com/facebookresearch/OrienterNet.