  In this work we apply and expand on a recently introduced outlier detection
algorithm that is based on an unsupervised random forest. We use the algorithm
to calculate a similarity measure for stellar spectra from the Apache Point
Observatory Galactic Evolution Experiment (APOGEE). We show that the similarity
measure traces non-trivial physical properties and contains information about
complex structures in the data. We use it for visualization and clustering of
the dataset, and discuss its ability to find groups of highly similar objects,
including spectroscopic twins. Using the similarity matrix to search the
dataset for objects allows us to find objects that are impossible to find using
their best fitting model parameters. This includes extreme objects for which
the models fail, and rare objects that are outside the scope of the model. We
use the similarity measure to detect outliers in the dataset, and find a number
of previously unknown Be-type stars, spectroscopic binaries, carbon rich stars,
young stars, and a few that we cannot interpret. Our work further demonstrates
the potential for scientific discovery when combining machine learning methods
with modern survey data.
