Learning and Deploying Local Features

Abstract:

Many computer vision applications including image matching, image-based reconstruction and localization rely on extracting and matching robust local features. A
typical local feature pipeline first detects repeatable keypoints in the image (i.e., keypoint detector), and then computes a short vector to uniquely describe each keypoint
(i.e., feature descriptor). Both the keypoint detector and the feature descriptor are
conventionally hand-crafted based on what is intuitive to the designer. For example, corners or blobs are popular choices of keypoints, and the image gradient is a
useful clue for descriptors. However, it is often dicult to define these principles to
accommodate various applications. In this thesis, we study data-driven approaches
which can more easily tailor the local feature pipeline for target applications. We start
with a mobile robotics application that leverages local features extracted from ground
texture images to achieve high-precision global localization. The second part of the
thesis addresses the problem that existing keypoint detectors that are optimized for
natural images su↵er from sub-optimal performance on texture images. We therefore
learn a keypoint detector specifically for each type of texture using a deep neural network. Our detector automatically learns to identify keypoints that are distinctive in
the target texture rather than relying on a set of pre-defined rules. Finally, we focus
on a non-parametric approach for learning feature descriptors. Many well-performing
local feature descriptors are trained using a triplet loss that includes a tunable margin, which limits its ability to generalize to other types of data and problems. We
propose to replace the hard margin with a soft margin that self-tunes as learning
progresses. To summarize, we first demonstrate through a novel visual-based localization system where a customized local feature pipeline is critical. Then, we tackle
both the keypoint detector and the feature descriptor with generalizable data-driven
approaches.