Local Analysis for 3D Reconstruction
of Specular Surfaces

Silvio Savarese, Min Chen and Pietro Perona

California Institute of Technology

This work is supported by NSF:
"3D Perception of Specular Surface" [NSF grant #0413312]

 

 

 

 

 

 

 

 

 

Figure 1. A curved mirror produces ‘distorted’ images of the surrounding world. We explore the geometry linking the shape of a curved mirror surface to the distortions produced on a scene it reflects.

Abstract. We study the problem of recovering the 3D shape of a smooth specular surface from a single image. First, we find the analytical relationship linking the local shape of a curved mirror surface to the distortions produced on a scene it reflects. We call this step direct map. Second, by taking advantage of this analytical relationship, we demonstrate that local shape of the surface may be recovered (up to third order) by measuring the local deformation of the reflected image of a planar scene pattern. We call this step inverse map or 3D recovery step. In the inverse map, we assume that the surface is unknown, the camera is calibrated and the geometry and position of the scene pattern is known in the camera reference system. We validate our theoretical results with experiments with real surfaces.

Introduction and motivation. Estimating the 3D shape of physical objects is one of most useful functions of vision. Texture, shading, contour, stereoscopy, motion parallax and active projection of structured lighting are the most frequently studied cues for recovering 3D shape. These cues, however, are often inadequate for recovering the shape of shiny reflective objects, such as a silver plate, a glass goblet or a well-washed automobile, for it is not possible to observe their surfaces directly, rather only what they reflect.

Yet, the ability of recovering the shape of specular or highly reflective surfaces is valuable in many applications such as industrial metrology of polished parts, medical imaging of moist or gelatinous tissues, digital archival of artworks and heritage objects, remote sensing of liquid surfaces, and diagnostic of space metallic structures, among others. See figure 2.

Although specular surfaces are difficult to measure with traditional techniques, specular reflections present an additional cue that potentially may be exploited for shape recovery. A curved mirror produces ‘distorted’ images of the surrounding world (see figure 1a). For example, the image of a straight line reflected by a curved mirror is, in general, a curve. It is clear that such distortions are systematically related to the shape of the surface. Is it possible to invert this map, and recover the shape of the mirror from its reflected images? The general ‘inverse mirror’ problem is clearly under-constrained: by opportunely manipulating the surrounding world, we may produce a great variety of images from any curved mirror surface, as illustrated by the anamorphic images that were popular during the Renaissance (see Figure 1b). This inverse problem may become tractable under the assumption that the structure of the scene is known.

Figure 2. The ability of recovering the shape of specular or highly reflective surfaces is valuable in many applications.

Goals. First, we find the analytical relationship linking the local shape of a curved mirror surface to the distortions produced on a scene it reflects. We call this step direct map. Second, by taking advantage of this analytical relationship, we demonstrate that local information about the geometry of the surface may be recovered by measuring the local deformation of the reflected images of a planar scene pattern. We call this step inverse map or 3D recovery step. In the inverse map, we assume that the surface is unknown, the camera is calibrated and the geometry and position of the scene pattern is known in the camera reference system. See Figure 3 - left.

Direct map . We start from the observation that the mapping from a scene line to its reflected curve in the camera image plane (due to mirror reflection) not only changes orientation of the scene line, but also "stretches” its length, modifying the local scale of the scene line. Such a deforming mapping can be easily illustrated by the grid points (1, 2, 3) and their corresponding reflected points (1', 2', 3') on the mirror surface. See figure 3, right. We first analyze this direct map and derive analytical expressions for the local geometry in the image (namely, orientation and local scale of reflected image curve at a given point - e.g. 1') as a function of the position and shape of the mirror surface. We then generalize our results to a planar scene patch of arbitrary texture: we describe the direct mapping between the scene patch (around a given point po) to the corresponding reflected patch in the image plane (around qo) as function of the local parameters of the mirror surface (around ro). Here the point ro is the reflection of po and qo is its image. See figure 4.

Figure 3. Left: setup: A (calibrated) camera is observing the planar scene reflected on the surface of the smooth mirror surface. Right: mapping between scene points 1, 2, 3, 4, 5 and their corresponding reflections 1', 2', 3', 4', 5'.

 

Figure 4. The direct mapping: we found the analytical expression describing the mapping from an arbitrary planar patch in the scene (around a generic point po) to the corresponding specular reflection in the image. The mapping is function of the local geometry of the mirror surface around the reflection point ro.

Inverse map - shape recovery. We measure local position, orientation and local scale of a reflected planar scene patch in the image plane. For instance, such local image measurements may be computed at a point (e.g. 1' in figure 3-right, or qo in figure 4) from its four neighboring reflected points (e.g. 2' , 3' , 4' , 5' ). By comparing these measurements with their corresponding analytical expressions, we induce a set of constraints, which lead to the solution for the surface position ro. Once ro is found, normal, curvature and third-order local parameters of the surface around ro can be found in closed-form solution.

Experimental Results. We validated our theoretical results by recovering local surface parameters of real mirror objects. A Kodak DC290 digital camera with 1792 × 1200 resolution was used to capture an image of a mirror surface reflecting a planar grid. The corners of the grid served as mark points and were used to measure position, orientation and local scale of reflected grid. Notice the grid can be replaced by any pattern as long as local position, orientation and scale can be extracted from the reflected image.

We first validated our 3D reconstruction technique with three simple shapes (plane, sphere and cylinder) where ground truth measurements were available. See figure 5 and table 1. Then we carried out experiments with "real world" surfaces such as a specular teapot and a portion of car fender. See figure 6.

Figure 5. Left a. A cylindrical mirror is placed over the grid pattern. We reconstructed 20 surface points and normals. The reconstructed points are highlighted with white circles. Left b. Top view of the reconstruction. Each reconstructed surface point is depicted by its tangent plane and normal. Right a. A spherical mirror placed over the grid pattern. Right b. Top view of the reconstruction.

 

Figure 6. Left. A tea pot is placed on the side of a grid pattern. We reconstructed the surface points and normals highlighted with black circles. Right. 45 degree view of the reconstruction.

 

Figure 7. Reconstruction of the car fender. Notice that reconstruction is feasible with only one image: the visible portion of the checkerboard is used the calibrate the camera and scene. The reflection of the checkerboard on the surface is used to recover the local shape of car fender.

 

Conclusions. Our main contribution are:

Our method departs and improves previous work in that: i) we work on the static monocular case (as shown in figure 7, with only one image we can calibrate the camera, the scene and recover information about an unknown specular shape); ii) our reconstruction scheme requires a simple setup and minimal hardware: a digital camera and a planar textured board; iii) no initial conditions about the surface position and shape are required iv) local shape can be recovered in closed from solution; v) our analysis is local and differential rather than global and algebraic.

We view our analysis as a promising start in the quest of computing the global shape of specular surfaces under fairly general conditions. The case of an uncalibrated world appears much more challenging and will require most certainly the integration of additional cues and some form of prior knowledge on the likely statistics of the scene geometry.

 

REFERENCES

Suggested reading:

S. Savarese, M. Chen and P. Perona, "Local Shape from Mirror Reflections",International Journal of Computer Vision (IJCV), 64(1), 31–67, 2005, - PDF

Addional readings:

S. Savarese, M. Chen and P. Perona, "Recovering local shape of a mirror surface from reflection of a regular grid", in Proc. of European Conference of Computer Vision , Prague , May 2004 - PDF

S. Savarese and P. Perona, "Local Analysis for 3D Reconstruction of Specular Surfaces -- part II", in Proc. of 7th European Conference of Computer Vision , Denmark , May 2002. PDF - Webpage

S. Savarese and P. Perona, "Local Analysis for 3D Reconstruction of Specular Surfaces", in Proc. of IEEE Conference on Computer Vision and Pattern Recognition , Kawa'i, USA, December 2001. PDF
Web page:

S. Savarese, M. Chen and P. Perona, "Second Order Local Analysis for 3D Reconstruction of Specular Surfaces", in Proc of 1st International Symposium on 3D Data Processing, Visualization and Transmission , IEEE Press, 2002 Abstract

"Local geometry of specular surfaces part I & II" (2002 ECCV, 2001 CVPR papers)

"On Local Properties of the Geometry of Specular Reflections" - a journey through the specular reflection world!

 

 

 

LINKS

Links to recent projects on 3D reconstuction of specular surfaces:

Voxel Carving for Specular Surfaces - T. Bonfort and P. Sturm

Shape from Distortion: 3D Range Scanning of Mirroring Objects - M. Tarini , H.Lensch, M.Gösle, HP.Seidel

Reconstructing Curved Surfaces From Specular Reflection Patterns - M. Halstead, B. Barsky, S. Klein R. Mandell

PDE Based Shape from Specularities - J. Solem, H. Aanæs , A. Heyden

Acquiring a Complete 3D Model from Specular Motion - J.Y. Zheng and A. Murata

Structured Highlight Inspection of Specular Surfaces - A.S. Sanderson, L.E. Weiss, S.K. Nayar

Phase Measuring Deflectometry - M. C. Knauer, J. Kaminski and G. Hausler

On Refractive Optical Flow - S. Agarwal, S. Mallick, D. Kriegman, and S. Belongie