Silvio Savarese*, Marco Andreetto*, Holly Rushmeier **, Fausto Bernardini **, Pietro Perona* * California Institute of Technology ** IBM T.J. Watson Research Center
Abstract. The shape of an object may be estimated by observing the shadows on its surface. Assuming that a conservative estimate of the object shape is available, our method analyzes images of the object illuminated with known point light sources and taken from known camera locations. The surface estimate is adjusted using the shadow regions to produce a refinement that is still a conservative estimate. A proof of correctness is provided. The method has been tested and validated with experimental results.
Introduction and motivation. We introduce a method for using self-shadows to estimate shape of an object that can be implemented using inexpensive lighting and imaging equipment.We assume that we have as a starting point a conservative estimate of object shape - that is, the volume enclosed by the current surface estimate completely contains the physical object.We analyze images of the object illuminated with known point light sources taken from known camera locations. We adjust the current surface estimate using the shadow regions to produce improved shape estimates that remain conservative. A proof of correctness is provided. Shape from shadows has the advantage that it does not rely on surface texture for establishing correspondence, or on a model of the surface reflectance characteristics. However, past methods for shape from shadows can give either poor results or fail to converge when applied to physical data that contains error. Our method is robust with respect to a conservative classification of shadow regions - pixels in shadow may be mislabelled as being lit.No assumptions about the surface topology are made (multi-part objects and occlusions are allowed), although any non-smooth regions over the object's surface are supposed to be detected.Our motivation for pursuing this work is the construction of an inexpensive and robust scanning system based on shape from silhouettes and shape from shadows.Furthermore, the method can be located within a similar volumetric framework as that employed by traditional shape-from-silhouette and recent multiple view algorithms.
Setup and Geometry. Consider an object in 3-D space and a point light source L illuminating it. A certain number of shadows are cast over the object by parts of the object itself. The scene is observed by a calibrated camera with center Oc. See figure below.
In order to simplify the discussion we consider a slice of the scene. A slice is defined by one of the (infinite possible) planes defined by the light source and the center of the camera. Thus, now we talk about image line, object's area and contour rather than image plane, object's surface and volume. The extension to the 3D case is immediate by observing that the slice may sweep the entire object's volume.
The method. We start from a conservative estimate of the object countour (upper bound estimate). See figure below. The object is depicted in brown whereas its upper bound is in yellow. The light source L illuminates the object and cast a shadow S on its surface. Let us suppose that we are able to process the image in order to separate the shadows from the rest of the observed object. We call Si the observed shadow. What do the observed object self-shadows tell us about the real surface? The main information comes from the inconsistencies between the shadows which would be produced by the estimated surface and the observed shadows which are actually produced by the real surface. Thus, we could remove (carve out) regions from the current object estimate in order to reduce the inconsistencies, and therefore incrementally compute better estimates of the object's shape, while making sure that at each step an upper bound estimate is maintained (conservative carving). We call this procedure shadow carving because we carve out areas from the object by using observed shadows. Here is an example of how to find a carvable area:
Thus, the carvable area can be removed from the current upper bound estimate of the object's contour, generating an improved conservative approximation.
- The observed shadow Si and the center of the camera define the green sector in figure.
- Call Se the projection of the observed shadow Si into the current upper bound object estimate
- The light source and Se defines the pink sector in figure.
- The intersection between the green sector, the pink sector and the current upper bound object area (in yellow) give the carvable area (in red).
The process can be iterated by using multiple light sources. As the the light source changes, different shadows are cast over object contour. Thus, for each light source position, different carvable areas can be found and removed, producing incrementally better estimates of the object's shape. See figure below.
We provide a proof of correctness. Namely, we show that the carvable area is well defined in the general case (i.e. complex surface topology, shadows occluded by other object parts, surfaces with low albedo or high specular regions) and always lying outside the actual object. See the technical paper for details.
Results. The method has been implemented and tested in the simple trial system shown below. Using each of several lights in turn, and a camera in front, allows multiple sets of shadow data to be obtained for each object position.
Some results of the carving process are shown below. Upper panels: two views of the original object. Middle panels: reconstruction by using space carving only. Lower panels: reconstruction by shadow carving. Notice shadow carving successfully recovers the concavities which cannot be handled by space carving.
Conclusions. We presented an analysis of the problem of refining a conservative estimate of an objects shape by observing the shadows on the object when it is lit by a known point light source. We showed that a well defined portion of volume can be removed from the current object estimate. We proved a theorem that guarantees that when this portion is carved away from the shape, the shape still remains conservative. We demonstrated that this insight leads to an algorithm that can work correctly on real images. We called this method shadow carving.
Shadow carving improves previous work on shape from shadow in that it is more robust with respect to the classification of shadow regions and is not restricted to 2.5D terrain surfaces, but rather it may be applied to measuring the objects in the round. In order to validate our theory, we have implemented a reconstruction system that combines information from silhouettes and shadows. The new system uses inexpensive digital cameras and lamps. Our experiments with real and synthetic objects confirms that the property of conservative carving is achievable in practice and show that shadow carving produces a much better surface estimate than shape from silhouettes alone. Future research directions include finding the optimal configuration of lights and cameras that maximizes the amount of volume that can be carved away at each iteration.
S. Savarese, M. Andreetto, H. Rusmeier, F. Bernardini, P. Perona, “3D Reconstruction by Shadow Carving: Theory and Practical Evaluation”, International Journal of Computer Vision (IJCV) , vol 71, no. 3, pp. 305-336, March 2007.
S. Savarese, Holly Rushmeier, Fausto Bernardini and P. Perona, "Shadow Carving", in Proc. of the Int. Conf. on Computer Vision, Vancouver, Canada, June 2001.
S. Savarese, H. Rushmeier, F. Bernardini and P. Perona, "Implementation of a Shadow Carving System for Shape Capture", in Proc. of 1st International Symposium on 3D Data Processing, Visualization and Transmission, IEEE Press, 2002, pp. 107-114.