Local Analysis for 3D Reconstruction
of Specular Surfaces
Silvio Savarese, Min Chen and Pietro Perona 
California Institute of Technology This work is supported by NSF:

Figure 1. A curved mirror produces ‘distorted’ images of the surrounding world. We explore the geometry linking the shape of a curved mirror surface to the distortions produced on a scene it reflects. 
Introduction and motivation. Estimating the 3D shape of physical objects is one of most useful functions of vision. Texture, shading, contour, stereoscopy, motion parallax and active projection of structured lighting are the most frequently studied cues for recovering 3D shape. These cues, however, are often inadequate for recovering the shape of shiny reflective objects, such as a silver plate, a glass goblet or a wellwashed automobile, for it is not possible to observe their surfaces directly, rather only what they reflect.
Yet, the ability of recovering the shape of specular or highly reflective surfaces is valuable in many applications such as industrial metrology of polished parts, medical imaging of moist or gelatinous tissues, digital archival of artworks and heritage objects, remote sensing of liquid surfaces, and diagnostic of space metallic structures, among others. See figure 2.
Although specular surfaces are difficult to measure with traditional techniques, specular reflections present an additional cue that potentially may be exploited for shape recovery. A curved mirror produces ‘distorted’ images of the surrounding world (see figure 1a). For example, the image of a straight line reflected by a curved mirror is, in general, a curve. It is clear that such distortions are systematically related to the shape of the surface. Is it possible to invert this map, and recover the shape of the mirror from its reflected images? The general ‘inverse mirror’ problem is clearly underconstrained: by opportunely manipulating the surrounding world, we may produce a great variety of images from any curved mirror surface, as illustrated by the anamorphic images that were popular during the Renaissance (see Figure 1b). This inverse problem may become tractable under the assumption that the structure of the scene is known.
Figure 2. The ability of recovering the shape of specular or highly reflective surfaces is valuable in many applications. 
Goals. First, we find the analytical relationship linking the local shape of a curved mirror surface to the distortions produced on a scene it reflects. We call this step direct map. Second, by taking advantage of this analytical relationship, we demonstrate that local information about the geometry of the surface may be recovered by measuring the local deformation of the reflected images of a planar scene pattern. We call this step inverse map or 3D recovery step. In the inverse map, we assume that the surface is unknown, the camera is calibrated and the geometry and position of the scene pattern is known in the camera reference system. See Figure 3  left.
Direct map . We start from the observation that the mapping from a scene line to its reflected curve in the camera image plane (due to mirror reflection) not only changes orientation of the scene line, but also "stretches” its length, modifying the local scale of the scene line. Such a deforming mapping can be easily illustrated by the grid points (1, 2, 3) and their corresponding reflected points (1', 2', 3') on the mirror surface. See figure 3, right. We first analyze this direct map and derive analytical expressions for the local geometry in the image (namely, orientation and local scale of reflected image curve at a given point  e.g. 1') as a function of the position and shape of the mirror surface. We then generalize our results to a planar scene patch of arbitrary texture: we describe the direct mapping between the scene patch (around a given point po) to the corresponding reflected patch in the image plane (around qo) as function of the local parameters of the mirror surface (around ro). Here the point ro is the reflection of po and qo is its image. See figure 4.
Figure 3. Left: setup: A (calibrated) camera is observing the planar scene reflected on the surface of the smooth mirror surface. Right: mapping between scene points 1, 2, 3, 4, 5 and their corresponding reflections 1', 2', 3', 4', 5'. 
Figure 4. The direct mapping: we found the analytical expression describing the mapping from an arbitrary planar patch in the scene (around a generic point po) to the corresponding specular reflection in the image. The mapping is function of the local geometry of the mirror surface around the reflection point ro.

Inverse map  shape recovery. We measure local position, orientation and local scale of a reflected planar scene patch in the image plane. For instance, such local image measurements may be computed at a point (e.g. 1' in figure 3right, or qo in figure 4) from its four neighboring reflected points (e.g. 2' , 3' , 4' , 5' ). By comparing these measurements with their corresponding analytical expressions, we induce a set of constraints, which lead to the solution for the surface position ro. Once ro is found, normal, curvature and thirdorder local parameters of the surface around ro can be found in closedform solution.
Experimental Results. We validated our theoretical results by recovering local surface parameters of real mirror objects. A Kodak DC290 digital camera with 1792 × 1200 resolution was used to capture an image of a mirror surface reflecting a planar grid. The corners of the grid served as mark points and were used to measure position, orientation and local scale of reflected grid. Notice the grid can be replaced by any pattern as long as local position, orientation and scale can be extracted from the reflected image.
We first validated our 3D reconstruction technique with three simple shapes (plane, sphere and cylinder) where ground truth measurements were available. See figure 5 and table 1. Then we carried out experiments with "real world" surfaces such as a specular teapot and a portion of car fender. See figure 6.
Figure 5. Left a. A cylindrical mirror is placed over the grid pattern. We reconstructed 20 surface points and normals. The reconstructed points are highlighted with white circles. Left b. Top view of the reconstruction. Each reconstructed surface point is depicted by its tangent plane and normal. Right a. A spherical mirror placed over the grid pattern. Right b. Top view of the reconstruction. 
Figure 6. Left. A tea pot is placed on the side of a grid pattern. We reconstructed the surface points and normals highlighted with black circles. Right. 45 degree view of the reconstruction. 
Figure 7. Reconstruction of the car fender. Notice that reconstruction is feasible with only one image: the visible portion of the checkerboard is used the calibrate the camera and scene. The reflection of the checkerboard on the surface is used to recover the local shape of car fender. 
Conclusions. Our main contribution are:
Our method departs and improves previous work in that: i) we work on the static monocular case (as shown in figure 7, with only one image we can calibrate the camera, the scene and recover information about an unknown specular shape); ii) our reconstruction scheme requires a simple setup and minimal hardware: a digital camera and a planar textured board; iii) no initial conditions about the surface position and shape are required iv) local shape can be recovered in closed from solution; v) our analysis is local and differential rather than global and algebraic.
We view our analysis as a promising start in the quest of computing the global shape of specular surfaces under fairly general conditions. The case of an uncalibrated world appears much more challenging and will require most certainly the integration of additional cues and some form of prior knowledge on the likely statistics of the scene geometry.
REFERENCES
Suggested reading:
S. Savarese, M. Chen and P. Perona, "Local Shape from Mirror Reflections",International Journal of Computer Vision (IJCV), 64(1), 31–67, 2005,  PDF
Addional readings:
S. Savarese, M. Chen and P. Perona, "Recovering local shape of a mirror surface from reflection of a regular grid", in Proc. of European Conference of Computer Vision , Prague , May 2004  PDF
S. Savarese and P. Perona, "Local Analysis for 3D Reconstruction of Specular Surfaces  part II", in Proc. of 7th European Conference of Computer Vision , Denmark , May 2002. PDF  Webpage
S. Savarese and P. Perona, "Local Analysis for 3D Reconstruction of Specular Surfaces", in Proc. of IEEE Conference on Computer Vision and Pattern Recognition , Kawa'i, USA, December 2001.
PDF
Web page:
S. Savarese, M. Chen and P. Perona, "Second Order Local Analysis for 3D Reconstruction of Specular Surfaces", in Proc of 1st International Symposium on 3D Data Processing, Visualization and Transmission , IEEE Press, 2002 Abstract
"Local geometry of specular surfaces part I & II" (2002 ECCV, 2001 CVPR papers)
"On Local Properties of the Geometry of Specular Reflections"  a journey through the specular reflection world!
LINKS Links to recent projects on 3D reconstuction of specular surfaces: Voxel Carving for Specular Surfaces  T. Bonfort and P. Sturm Shape from Distortion: 3D Range Scanning of Mirroring Objects  M. Tarini , H.Lensch, M.Gösle, HP.Seidel Reconstructing Curved Surfaces From Specular Reflection Patterns  M. Halstead, B. Barsky, S. Klein R. Mandell PDE Based Shape from Specularities  J. Solem, H. Aanæs , A. Heyden Acquiring a Complete 3D Model from Specular Motion  J.Y. Zheng and A. Murata Structured Highlight Inspection of Specular Surfaces  A.S. Sanderson, L.E. Weiss, S.K. Nayar Phase Measuring Deflectometry  M. C. Knauer, J. Kaminski and G. Hausler On Refractive Optical Flow  S. Agarwal, S. Mallick, D. Kriegman, and S. Belongie 