Vision CNNs trained to estimate spatial latents learned similar ventral-stream-aligned representations
Yudi Xie, Weichen Huang, Esther Alter, Jeremy Schwartz, Joshua B. Tenenbaum, James J. DiCarlo
Abstract
Studies of the functional role of the primate ventral visual stream have traditionally focused on object categorization, often ignoring -despite much prior evidence -its role in estimating "spatial" latents such as object position and pose. Most leading ventral stream models are derived by optimizing networks for object categorization, which seems to imply that the ventral stream is also derived under such an objective. Here, we explore an alternative hypothesis: Might the ventral stream be optimized for estimating spatial latents? And a closely related question: How different -if at all -are representations learned from spatial latent estimation compared to categorization? To ask these questions, we leveraged synthetic image datasets generated by a 3D graphic engine and trained convolutional neural networks (CNNs) to estimate different combinations of spatial and category latents. We found that models trained to estimate just a few spatial latents achieve neural alignment scores comparable to those trained on hundreds of categories, and the spatial latent performance of models strongly correlates with their neural alignment. Spatial latent and category-trained models have very similar -but not identical -internal representations, especially in their early and middle layers. We provide evidence that this convergence is partly driven by non-target latent variability in the training data, which facilitates the implicit learning of representations of those non-target latents. Taken together, these results suggest that many training objectives, such as spatial latents, can lead to similar models aligned neurally with the ventral stream. Thus, one should not assume that the ventral stream is optimized for object categorization only. As a field, we need to continue to sharpen our measures of comparing models to brains to better understand the functional roles of the ventral stream.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on7
- WILDS: A Benchmark of in-the-Wild Distribution ShiftsPang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie et al.ICML 2021 · 1,773 citations
- In Search of Lost Domain GeneralizationIshaan Gulrajani, David Lopez-PazICLR 2021 · 1,416 citations
- Revisiting Model Stitching to Compare Neural RepresentationsYamini Bansal, Preetum Nakkiran, Boaz BarakNeurIPS 2021 · 253 citations
- StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation LearnersYonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang et al.NeurIPS 2023 · 251 citations
- Rosetta Neurons: Mining the Common Units in a Model ZooAmil Dravid, Yossi Gandelsman, Alexei A. Efros, Assaf ShocherICCV 2023 · 46 citations
Related papers
- Scaling Laws for Task-Optimized Models of the Primate Visual Ventral StreamAbdülkadir Gökce, Martin SchrimpfICML 2025
- Characterizing the Ventral Visual Stream with Response-Optimized Neural Encoding ModelsMeenakshi Khosla, Keith Jamison, Amy Kuceyeski, Mert R. SabuncuNeurIPS 2022 · 16 citations
- Sparse components distinguish visual pathways & their alignment to neural networksAmmar I Marvi, Nancy Kanwisher, Meenakshi KhoslaICLR 2025
- Dimensionality Mismatch Between Brains and Artificial Neural NetworksSantiago Galella, Maren H. Wehrheim, Matthias KaschubeNeurIPS 2025
- A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human BrainsMinkyu Choi, Kuan Han, Xiaokai Wang, Yizhen Zhang et al.NeurIPS 2023 · 33 citations
