Focal Plane Visual Feature Generation and Matching on a Pixel Processor Array

Pixel Processor Arrays (PPAs) are vision sensors that embed data and processing into every pixel element. PPAs can execute visual processing directly at the point of light capture, and output only sparse, high-level information. This is in sharp contrast with the conventional visual pipeline, in which whole images must be transferred from sensor to processor. This sparse data readout also provides several major benefits such as higher frame rate, lower energy consumption and lower bandwidth requirements. In this work, we demonstrate generation, matching and storage of binary descriptors for visual keypoint features, entirely upon PPA with no need to output images to external processing, mak- ing our approach inherently privacy-aware. Our method spreads descriptors across the memory of multiple pixel processors, allowing significantly larger descriptors than prior pixel-processing works. These larger descriptors can be used for a range of tasks such as place and object recognition. We demonstrate the accuracy of our in-pixel feature matching up to ∼94.5%, at ∼210fps, across a range of datasets, with a greater than 100× reduction in data transfer and bandwidth requirements over traditional cameras.