Nautilus Recovering Regional Symmetry Transformations for Image Editing
MICHAL LUKÁČ, Adobe Research and Czech Technical University in Prague
DANIEL SÝKORA, Czech Technical University in Prague
KALYAN SUNKAVALLI, Adobe Research
ELI SHECHTMAN, Adobe Research
ONDŘEJ JAMRIŠKA, Czech Technical University in Prague
NATHAN CARR, Adobe Research
TOMÁŠ PAJDLA, Czech Technical University in Prague
6 APPLICATIONS
To demonstrate the practical utility and versatility of our method, we have implemented several image editing applications where the knowledge of the symmetric transformation and its spatial support simplifies the task or removes the need for manual input. We compare our results on each of these applications to their current state-of-the-art methods, and demonstrate that our general framework is able to replicate and improve upon their quality.
6.1 Image Rectification
Previous techniques for removing perspective distortion and shear from a photograph require the manual specification or detection of corresponding points, right angles, vanishing lines [Liebowitz and Zisserman 1998], congruent line segments [Aiger et al. 2012], global measurements such as the rank of the image matrix [Zhang et al. 2012] or change of scale [Pritts et al. 2014].
In our solution we rectify the image using one of our detected global symmetry homographies, ${\prosedeflabel{nautilus}{{H}}}$ . The key is to assume that ${\proselabel{nautilus}{{H}}}$ has the following structure:
${\prosedeflabel{nautilus}{{P}}}$ is the pure perspective part of ${\proselabel{nautilus}{{H}}}$ and ${\prosedeflabel{nautilus}{{S}}}$ is an in-plane symmetry which we assume to be a general similarity transform, i.e., uniform scale, rotation, and translation. In addition, we assume that this similarity transformation has a non-zero rotational component, failing which the decomposition can become under-constrained.
To obtain the rectified image we need to estimate ${\proselabel{nautilus}{{P}}}$ and then apply its inverse. To do that we formulate the task as a non-linear optimization problem, with the following objective function:
The first term of (10) enforces ${\proselabel{nautilus}{{S}}}$ to be as close to a similarity as possible, and the second term ensures that the decomposition is close to the input homography ${\proselabel{nautilus}{{H}}}$ (measured using the re-projection error of the four image corners ${\proselabel{nautilus}{{v}}}_{1...4}$).
Minimizing ${\proselabel{nautilus}{{E}}}({\proselabel{nautilus}{{x}}})$ is feasible only when the rotation angle of the underlying similarity is away from 0◦ and 180◦. This can be verified by finding the closest similarity ${\proselabel{nautilus}{{S}}}^′$ to the input homography ${\proselabel{nautilus}{{H}}}$ and computing $α = arctan({\proselabel{nautilus}{{x}}}₈/{\proselabel{nautilus}{{x}}}₅)$. When $α ∈ (10◦, 170◦)$ we initialize ${\proselabel{nautilus}{{P}}}$ with the identity matrix, ${\proselabel{nautilus}{{S}}} = {\proselabel{nautilus}{{S}}}^′$ and then run a non-linear optimization using the L-BFGS algorithm [Liu and Nocedal 1989; Nocedal 1980] where exact partial derivatives of ${\proselabel{nautilus}{{E}}}({\proselabel{nautilus}{{x}}})$ are computed using dual numbers [Piponi 2004]. To avoid getting stuck in an inappropriate local minima, we allow for greater freedom at the beginning of the optimization. We start with ${\proselabel{nautilus}{{λ}}} = 1$ to let the algorithm explore a fruitful direction, and gradually increase it until it reaches ${\proselabel{nautilus}{{λ}}} = 109$ which allows us to strictly enforce ${\proselabel{nautilus}{{S}}}$ to be a similarity transform. To ensure that the resulting decomposition is meaningful we use the value of ${\proselabel{nautilus}{{E}}}({\proselabel{nautilus}{{x}}})$. In practice, we observed ${\proselabel{nautilus}{{E}}}({\proselabel{nautilus}{{x}}}) < 1$ indicates a good rectification (see Fig. 8).
