Continuous learning of inverse problems with applications to structure from motion

This thesis deals with the problem of improving classical methods for scene reconstruction via multi-view geometry, by combining them with modern, deep learning techniques. This is related to the problem of taking inverse problems, solved by minimising an energy function, and including them into pipelines containing deep network elements. Ideally, training these pipelines end-to-end should allow benefiting from the best of both worlds, combining the flexibility and ability to learn problem solving from data of deep networks, and the carefully designed and easily interpretable energy functions of classical inverse problems. The main issue, however, is that such inverse problems are often non-differentiable, as their solution is obtained by solving an argmin problem.

This thesis makes four main contributions:(i)the first involves improving the energy function of a model in which the inverse problem admits a closed-form-solution and is therefore differentiable; since this is not the case for most inverse problems,(ii)the second contribution is to formulate a general framework, called “Wibergian learning”, that allows computing the gradient from the energy optimisation output independently of the optimiser. This process is reminiscent of implicit differentiation but is stable for a larger class of problems. (iii)The next contribution is using this framework to turn the non-differentiable Bundle Adjustment (BA) component of a preexisting pipeline into a single differentiable layer, and(iv)investigating whether such a layer may be included in a different pipeline, containing deep networks and trained end-to-end.

The first three contributions were successfully carried out, and this thesis provides experimental results to validate them. The pipelines introduced in contribution (iv) were less successful, showing poor generalisation. However, ablative studies suggest that such limitations are likely due to the training data and the chosen deep networks, and not to the BA layer. Moreover, such experiments still provide important information about the generalisation gap and the main source of error in similar pipelines.

Overall, these contributions and the corresponding experiments show how geometry-based inverse problems can be significantly improved by combining them with deep learning tools, and how the framework introduced at point (ii) can successfully be applied to pre-existing models.

Attend the event

All are welcome to attend the event on Zoom.