  Invoking the manifold assumption in machine learning requires knowledge of
the manifold's geometry and dimension, and theory dictates how many samples are
required. However, in applications data are limited, sampling may not be
uniform, and manifold properties are unknown and (possibly) non-pure; this
implies that neighborhoods must adapt to the local structure. We introduce an
algorithm for inferring adaptive neighborhoods for data given by a similarity
kernel. Starting with a locally-conservative neighborhood (Gabriel) graph, we
sparsify it iteratively according to a weighted counterpart. In each step, a
linear program yields minimal neighborhoods globally and a volumetric statistic
reveals neighbor outliers likely to violate manifold geometry. We apply our
adaptive neighborhoods to non-linear dimensionality reduction, geodesic
computation and dimension estimation. A comparison against standard algorithms
using, e.g., k-nearest neighbors, demonstrates their usefulness. Code for our
algorithm will be available at https://github.com/dyballa/IAN
