  A fundamental problem in supervised learning is to find a good set of
features or distance measures. If the new set of features is of lower
dimensionality and can be obtained by a simple transformation of the original
data, they can make the model understandable, reduce overfitting, and even help
to detect distribution drift. We propose a supervised dimensionality reduction
method Gradient Boosting Mapping (GBMAP), where the outputs of weak learners --
defined as one-layer perceptrons -- define the embedding. We show that the
embedding coordinates provide better features for the supervised learning task,
making simple linear models competitive with the state-of-the-art regressors
and classifiers. We also use the embedding to find a principled distance
measure between points. The features and distance measures automatically ignore
directions irrelevant to the supervised learning task. We also show that we can
reliably detect out-of-distribution data points with potentially large
regression or classification errors. GBMAP is fast and works in seconds for
dataset of million data points or hundreds of features. As a bonus, GBMAP
provides a regression and classification performance comparable to the
state-of-the-art supervised learning methods.
