Shadow Carving

 
Silvio Savarese*, Marco Andreetto*,  Holly Rushmeier **, Fausto Bernardini **, Pietro Perona*
* California Institute of Technology
** IBM T.J. Watson Research Center

Abstract.
Cast shadows are an informative cue to the shape of objects. They are particularly useful for discovering object’s concavities. We propose a new method for recovering shape from cast shadows which we call shadow carving. Given a conservative estimate of the volume occupied by an object this method computes a new estimate which is consistent with the observed pattern of shadows. This new estimate is provably conservative, i.e. the shadow carving never removes portion of the real object. We propose a reconstruction system to recover shape from silhouettes and shadow carving. The silhouettes are used to reconstruct the initial conservative estimate of the object’s shape and shadow carving is used to carve out the concavities. We have simulated our reconstruction system with a commercial rendering package to explore the design parameters and assess the accuracy of the reconstruction. We have also implemented our reconstruction scheme in a table-top system and present the results of scanning of several objects.

Introduction and motivation.
We introduce a method for using self-shadows to estimate shape of an object that can be implemented using inexpensive lighting and imaging equipment.We assume that we have as a starting point a conservative estimate of object shape - that is, the volume enclosed by the current surface estimate completely contains the physical object.We analyze images of the object illuminated with known point light sources taken from known camera locations. We adjust the current surface estimate using the shadow regions to produce improved shape estimates that remain conservative. A proof of correctness is provided. Shape from shadows has the advantage that it does not rely on surface texture for establishing correspondence, or on a model of the surface reflectance characteristics. However, past methods for shape from shadows can give either poor results or fail to converge when applied to physical data that contains error. Our method is robust with respect to a conservative classification of shadow regions - pixels in shadow may be mislabelled as being lit.No assumptions about the surface topology are made (multi-part objects and occlusions are allowed), although any non-smooth regions over the object's surface are supposed to be detected.Our motivation for pursuing this work is the construction of an inexpensive and robust scanning system based on shape from silhouettes and shape from shadows.Furthermore, the method can be located within a similar volumetric framework as that employed by traditional shape-from-silhouette and recent multiple view algorithms.

Setup and Geometry.
Consider an object in 3-D space and a point light source L illuminating it. A certain number of shadows are cast over the object by parts of the object itself. The scene is observed by a calibrated camera with center Oc. See figure below.


In order to simplify the discussion we consider a slice of the scene. A slice is defined by one of the (infinite possible) planes defined by the light source and the center of the camera. Thus, now we talk about image line, object's area and contour rather than image plane, object's surface and volume. The extension to the 3D case is immediate by observing that the slice may sweep the entire object's volume.



The method.
We start from a conservative estimate of the object countour (upper bound estimate). See figure below. The object is depicted in brown whereas its upper bound is in yellow. The light source L illuminates the object and cast a shadow S on its surface. Let us suppose that we are able to process the image in order to separate the shadows from the rest of the observed object. We call Si the observed shadow. What do the observed object self-shadows tell us about the real surface? The main information comes from the inconsistencies between the shadows which would be produced by the estimated surface and the observed shadows which are actually produced by the real surface. Thus, we could remove (carve out) regions from the current object estimate in order to reduce the inconsistencies, and therefore incrementally compute better estimates of the object's shape, while making sure that at each step an upper bound estimate is maintained (conservative carving). We call this procedure shadow carving because we carve out areas from the object by using observed shadows. Here is an example of how to find a carvable area:

  1. The observed shadow Si and the center of the camera define the green sector in figure.
  2. Call Se the projection of the observed shadow Si into the current upper bound object estimate
  3. The light source and Se defines the pink sector in figure.
  4. The intersection between the green sector, the pink sector and the current upper bound object area (in yellow) give the carvable area (in red).
Thus, the carvable area can be removed from the current upper bound estimate of the object's contour, generating an improved conservative approximation.


The process can be iterated by using multiple light sources. As the the light source changes, different shadows are cast over object contour. Thus, for each light source position, different carvable areas can be found and removed, producing incrementally better estimates of the object's shape. See figure below.


We provide a proof of correctness. Namely, we show that the carvable area is well defined in the general case (i.e. complex surface topology, shadows occluded by other object parts, surfaces with low albedo or high specular regions) and always lying outside the actual object. See the technical paper for details.

Results.
The method has been implemented and tested with the setup shown below. In all the experiments the turntable completed a rotation in 24 steps, and 5 images of the object were acquired at each rotation step (one for each light in the array). Also a range image was acquired at each rotation step. The 24 range images were merged into a single 3D model used as reference for the true shape of the object.




Several 3D object were acquired using the proposed setup. The results for two such objects, a sculpture of a dog and a sculpture of a fireman, shown in the figures below. The first two figures refers to the dog sculpture. The left image in the first figure is a picture of the actual object. The central image is the initial upper bound estimate of the object (achieved by standard shape-from-silhouttes technique). The right image shows the object after carving using shadows. Altough the reconstruction is still coarse, the concavity between the two leg of the dog  has become visible.



The second figure below show the sections of the real object (red), the initial upper bound (blue), and the final estimate form shadow carving (black), all referred to the same reference point. The sections on the left are obtained after carving with 6 rotation steps. The central ones after 18 steps, and the right one is the final estimate obtained with 24 steps. It is clearly visible how portions of volume are removed from the current estimate which more and more approaching the section of the real object. Also the shadow carving estimate is always outside the volume of the real object, i.e. it is always a conservative estimate of the object volume.



The second object, the fireman is presented below. Each row refer to a different acquisition viewpoint. As for the dog sculpture a picture of the real object, the initial estimate and the final shadow carving estimate are proposed. The last image in each row presents the shadow carving estimate with superimpose a texture. Using the texture, small scale geometric details can be visually reproduced.



Publications.

M. Andreetto, S. Savarese, and P. Perona, “Carving from Ray-tracing Constraints: IRT-carving”, Third International Symposium on 3D Data Processing Visualization and Transmission (3DPVT06), Chapel Hill, USA, Jun. 2006. (PDF)

S. Savarese, M. Andreetto, H. Rushmeier, F. Bernardini, and P. Perona, “3D Reconstruction by Shadow Carving: Theory and Practical Evaluation”, International Journal of Computer Vision (IJCV), vol 71, no. 3, pp. 305-336, March 2007. (PDF)

S. Savarese, H. Rushmeier, F. Bernardini and P. Perona, “Implementation of a Shadow Carving System for Shape Capture”, in Proc. of 1st International Symposium on 3D Data Processing, Visualization and Transmission, IEEE Press, 2002, pp. 107-114. (PDF)

S. Savarese, Holly Rushmeier, Fausto Bernardini and P. Perona, “Shadow Carving”, in Proc. of the Int. Conf. on Computer Vision, Vancouver, Canada, June 2001. (PDF)