Silvio Savarese fffffMin Chen fffffPietro Perona
1. Single-point ambiguity. Let us consider a generic mirror surface and a scene composed by only one point xp. The scene point is reflected by the mirror surface (at xm) and imaged into the image plane of the camera (whose center is Oc), yielding the observation xi. See Fig. 1. It is not difficult to conclude that there exists a family of surfaces passing through xm(s) = s Xi, whose normal nm(s) at xm is given by Eq.1, that still produces the same observation Xi.
f is a vector function defined in http://www.vision.caltech.edu/savarese/local_analysis.html (figure 3). See   for further details. We call such result the "single-point" ambiguity. Click on the image below to download the movie. The movie shows a sequence of tangent planes moving along the line of sight, whose normals are parametrized by s and given by equation (1), producing the same observation Xi.
2. The tangent of a reflected line is function of first and second local surface parameters only. Let us now consider a line through xp. The line is reflected off the surface and yields a image curve passing through xi within the image plane. See Fig. 2.
It turns out that the orientation of the tangent of such line at xi is function of the second order surface parameters a,b,c -- namely, the second order parameters of the taylor expansion of the surface around the reflecting point xm:
(see http://www.vision.caltech.edu/savarese/local_analysis.html (figure 4) or   for details.
In order to illustrate such result, as an example, we consider a triplet of scene lines passing through xp. See Fig. 3 (left). Such lines are reflected off the mirror surface M. The corresponding image reflected curves appear in bold (red, green and blue colors) in Fig. 3 (right). For each curve, we compute and plot the corresponding tangent at xi. See solid straight lines in Fig. 3 (left). Then we consider a sequence of mirror surfaces. Each surface passes through xm and shares the same normal (and tangent plane) as the surface M. The sequence of surfaces is built by linearly decreasing the second order parameters a,b,c's only. See movie of Fig. 3 (left). For each surface we plot the corresponding image reflected curves. See movie of Fig.3 (right). As it appears clear from the movie, the tangents are function of the surface curvature: as the surface curvature changes, the orientations of the image reflected curves change as well. The dashes lines are the initial tangents, corresponding to surface M.
The movies in Fig. 4 show another interesting consequence of the theoretical analysis: the tangent orientations of image reflected curves do no depend on third or higher coefficients of the surface Taylor expansion (3). Again we consider a sequence of mirror surfaces. Each surface passes through xm and shares the same normal (and tangent plane) as the surface M. The sequence of surfaces is built by linearly changing the third order parameters e,f,g,h's only. See movie of Fig. 4 (left). For each surface we plot the corresponding image reflected curves. See movie of Fig. 4 (right). As appears clear from the movie, the tangents are invariant with respect the third order surface local parameters. Similar results can be obtained by changing 4th or higher surface parameters.
3. The First Order Ambiguity. Let us consider a generic mirror surface M and a set of N scene lines passing through xp. Each scene line is reflected off the surface and yields an image curve passing through xi within the image plane. Our observation vector T is the set of N orientations of the tangents of the corresponding N image curves at xi. Let sm be the distance between the camera center and the surface reflection point on M. Let us now consider the family of mirror surfaces passing through xm(s) = s xi (e.g. the line of sight) and having nornal nm(s) at xm given by Eq.1.We may want to address the following question: is M the only mirror surface yielding the observation vetctor T? Namely, are there other mirror surfaces belonging to such family that still produce the same observation vector T?
It turns out that if N=1 or 2, for each value of s, we can always find a mirror surface that produces same observation vector T ( by changing second order surface parameters a,b,c). However, if there are N>2 coplanar scene lines, such family of mirror surfaces may produce the same observation vector T for s equal to just a discrete number of values s1, s2, ... sj, ...sP (ghost solutions), besides the trivial solution s=sm. We call ghost mirror surfaces the corresponding surfaces M1,M2,... Mj,....Mp attached to s1, s2, ... sj, ...sP. See  for details. In other words, given a set of N>2 coplanar scene lines and a corresponding observation vector T produced by the mirror surface M, there might be a discrete set of mirror surfaces M1,M2,... Mj,....Mp that still produces the same observation vector T, Mj being the jth mirror surface which passes through xm (sj) and has normal nm(sj). We call such result, the "first order ambiguity". In Fig.5 there is an example of first order ambiguity: given a specular surface passing through xm(sm) and having normal nm(sm), there exists another specular surface passing through xm(s1) and having normal nm(s1) that produces the same observation vector T (same observed tangents).
By means of our theoretical derivation, given a set of scene lines and the corresponding observation vector T produced by the mirror surface M, it is possible to compute the actual solution sm and the complete set of ghost solutions s1, s2, ... sj, ...sP, and the corresponding mirror surfaces in term of first order description. From experimental analysis it turns out that, in general, there are either no ghost solutions at all or a very small number (i.e.max 2 or 3).
If there are N>3 no coplanar scene lines, the ghost solutions vanish -- it is not possible to find a mirror surface different from M (in term of first order description, i.e. position and normal) which is still consistent with the observation vector T produced by the mirror surface M.
4. The Second Order Ambiguity. Let us consider a generic mirror surface M and a set of N scene lines passing through xp. Let us suppose that the observation vector T defined in the previous section is available. In section 2, we have seen that the observed tangents are actually function of the second order surface local parameters (a,b,c) but not function of 3rd or higher order terms. Thus, we may wonder whether by changing second order surface paramaters only (i.e. while keeping fixed the first order ones) we can still obtain the same observation vector T. It turns out that there is a 1-parameter-family of mirror surfaces that produces an observation vector which is still consistent with T. Such 1-parameter-family of mirror surfaces has second order terms a,b,c parametrized as follows:
where a1, a2, b1, b2, c2 are known quantities and r is the parameter. See  for details. In other words, there exists a family of surfaces that passes through the reflection point xm, shares the same normal nm as M,has the second order local parameters described by Eq.4, has arbitrary third and higher local description and yet produces the same observation vector T. We call this result "second order ambiguity". We visually illustrate such ambiguity in the following figures and demos.
We first introduce a R3 space called the space of paraboloids, where the geometry describing our results can be represented in a more clear fashion. In the space of paraboloids, the coordinates of a point [a b c]T univocally describe a paraboloid given by:
See Fig. 6. For instance, the family of paraboloids described by Eq.4 is depicted as the straight dashed line. The locus c^2=ab (i.e. the set of all parabolic paraboloids) separates the space into two regions. All of the points such that c^2< ab correspond to elliptic paraboloids whereas all of the points such that c^2 >ab corresponds to hyperbolic paraboloids. Thus, if P1 and P2 are two paraboloids belonging to the same family, the former is an elliptic one and the latter an hyperbolic one. In Fig. 7 (left) P1 and P2 are shown in the XYZ reference system attached to the camera center. A scene line (red solid line) is considered. In Fig. 7 (right) the corresponding image reflected curves are plotted: the dashed green curve and the dashed red curve are generated by P1 and P2 respectively. Notice that the tangents of the curves at xio are coincident (solid blue line).
Click on the figures to show the demos. The demo attached to Fig. 6 shows the sequence of paraboloids along the family, as r increases. The sequence starts with a paraboloid attached to a small value of r (elliptic region) and ends with a paraboloid belonging to the parabolic region. The demo in Fig. 7 (left) shows the corresponding sequence of paraboloids in the XYZ reference system attached to the camera center. A triplet of scene lines is then considered and, for each paraboloid of the sequence, the corresponding image reflected curves are plotted, the tangents being depicted as solid lines. See movie of Fig. 7(right). From the movie we can notice that the tangents are invariant with respect to r: all of the paraboloids belonging the family defined in Eq.5 produces the same observation vector T (i.e. same tangents).
5. The curvature of the reflected line increases (asymptotically) linearly if the surface curvature increases linearly. By performing a second order local analysis , we can relate the curvature k of the image reflected curve at xio with the parameter r. It turns out that k is also function of the third order terms of the surface taylor expansion (3). We want to remark that k does not depend on 4th or higher terms. Assuming that the surface third order terms are negligable or known, it is possible to express the curvature k as function of r only. The relationship is of the type:
k = k1 1/r + k2 + k3 r ffffffffff(6)
where k1,k2,k3 are known quantities. See  for details. After computing the curvatures attached to the curves in demo of fig.5 (right), we plotted k as function of r. See Fig. 8. Click on the figure to watch the corresponding movie (values of k as r increases). Each color corresponds to the curvature of the image reflection curve of one of the scene lines. Notice how k becomes linear as r becomes large enough (i.e. asymptotically linear) according to equation (6).
6. The Third Order Ambiguity. We know from  that the curvature of the image reflected curve can be expressed as a linear combination of the third order surface parameters e,f,g,h of the Taylor expansion (3):
k = m(r) + n(r) [e f g h]T
where m and n are function of the second order parameters a,b,c and hence function of r (see Eq.4). Notice that n is a vector. By considering 4 lines (and therefore 4 curvature measurements k1,k2,k3,k4), we obtain the following system:
k1 = m1(r)
+ n1(r) [e f g h]T
Thus, in matrix form:
k = m(r) +
N(r) [e f g h]T
where N is a matrix. Thus,
[e f g h]T = N(r)-1 (k - m(r)) (7)
The demo below shows an example of third order ambiguity. Left demo: sequence of surfaces with second and third order surface parameters parameterized by r according to equations (4) and (7). Right demo: corresponding image reflected curves. Each of these surfaces yields image reflected curves with tangents T and curvatures K calculated at xio,
 S. Savarese, M. Chen and P. Perona, "Second Order Local Analysis for 3D Reconstruction of Specular Surfaces", in Proc. of 1st International Symposium on 3D Data Processing and Visualization , Italy, 2002.