Inverse lighting and photorealistic rendering for augmented reality

We present a practical and robust photorealistic rendering pipeline for augmented reality. We solve the real world lighting conditions from observations of a diffuse sphere or a rotated marker. The solution method is based on l₁-regularized least squares minimization, yielding a sparse set of light sources readily usable with most rendering methods. The framework also supports the use of more complex light source representations. Once the lighting conditions are solved, we render the image using modern real-time rendering methods such as shadow maps with variable softness, ambient occlusion, advanced BRDF’s and approximate reflections and refractions. Finally, we perform post-processing on the resulting images in order to match the various aberrations and defects typically found in the underlying real-world video.