EyeIR: Single Eye Image Inverse Rendering In the Wild

We propose a method to decompose a single eye region image in the wild into albedo, shading, specular, normal and illumination. This inverse rendering problem is particularly challenging due to inherent ambiguities and complex properties of the natural eye region. To address this problem, first we construct a synthetic eye region dataset with rich diversity. Then we propose a synthetic to real adaptation framework to leverage the supervision signals from synthetic data to guide the direction of self-supervised learning. We design region-aware self-supervised losses based on image formation and eye region intrinsic properties, which can refine each predicted component by mutual learning and reduce the artifacts caused by ambiguities of natural eye images. Particularly, we address the demanding problem of specularity removal in the eye region. We show high-quality inverse rendering results of our method and demonstrate its use for a number of applications.