CVPR Findings2026
M^3A Policy: Mutable Material Manipulation Augmentation Policy through Photometric Re-rendering
Jiayi Li, Yuxuan Hu, Haoran Geng, Xiangyu Chen, Chuhao Zhou, Ziteng Cui, Jianfei Yang
Abstract
Material generalization is essential for real-world robotic manipulation, where robots must interact with objects exhibiting diverse visual and physical properties. This challenge is particularly pronounced for objects made of glass, metal, or other materials whose transparent or reflective surfaces introduce severe out-of-distribution variations. Existing approaches either rely on simulated materials in simulators and perform sim-to-real transfer, which is hindered by substantial visual domain gaps, or depend on collecting extensive real-world demonstrations, which is costly, time-consuming, and still insufficient to cover various materials. To overcome these limitations, we resort to computational photography and introduce Mutable Material Manipulation Augmentation (MA), a unified framework that leverages the physical characteristics of materials as captured by light transport for photometric re-rendering. The core idea is simple yet powerful: given a single real-world demonstration, we photometrically re-render the scene to generate a diverse set of highly realistic demonstrations with different material properties. This augmentation effectively decouples task-specific manipulation skills from surface appearance, enabling policies to generalize across materials without additional data collection. To systematically evaluate this capability, we construct the first comprehensive multi-material manipulation benchmark spanning both simulation and real-world environments. Extensive experiments show that the MA policy significantly enhances cross-material generalization, improving the average success rate across three real-world tasks by 58.03%, and demonstrating robust performance on previously unseen materials.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- NeRD: Neural Reflectance Decomposition from Image Collections cites Neural Inverse Rendering of an Indoor Scene From a Single Image
- Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields cites NeRD: Neural Reflectance Decomposition from Image Collections
Builds on14
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- NeRD: Neural Reflectance Decomposition from Image CollectionsMark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron et al.ICCV 2021 · 608 citations
- Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance FieldsDor Verbin, Peter Hedman, Ben Mildenhall, Todd E. Zickler et al.CVPR 2022 · 477 citations
- Neural Inverse Rendering of an Indoor Scene From a Single ImageSoumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu et al.ICCV 2019 · 172 citations
Related papers
- CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous ManipulationJun Wang, Yuzhe Qin, Kaiming Kuang, Yigit Korkmaz et al.CVPR 2024 · 11 citations
- AOMGen: Photoreal, Physics-Consistent Demonstration Generation for Articulated Object ManipulationYulu Wu, Jiujun Cheng, Haowen Wang, Dengyang Suo, Pei Ren, Qichao Mao, Shangce Gao, Yakun HuangCVPR Findings 2026 · 3 citations
- Learning to See and Act: Task-Aware Virtual View Exploration for Robotic ManipulationYongjie Bai, Zhouxia Wang, Yang Liu, Kaijun Luo et al.CVPR 2026 · 6 citations
- Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control InterfaceYujie Zhao, Hongwei Fan, Di Chen, Shengcong Chen et al.CVPR 2026 · 8 citations
- Manipulation as in Simulation: Enabling Accurate Geometry Perception in RobotsMinghuan Liu, Zhengbang Zhu, Xiaoshen Han, Peng Hu et al.ICLR 2026 · 17 citations
