  The goal of dimension reduction tools is to construct a low-dimensional
representation of high-dimensional data. These tools are employed for a variety
of reasons such as noise reduction, visualization, and to lower computational
costs. However, there is a fundamental issue that is discussed in other
modeling problems that is often overlooked in dimension reduction --
overfitting. In the context of other modeling problems, techniques such as
feature-selection, cross-validation, and regularization are employed to combat
overfitting, but rarely are such precautions taken when applying dimension
reduction. Prior applications of the two most popular non-linear dimension
reduction methods, t-SNE and UMAP, fail to acknowledge data as a combination of
signal and noise when assessing performance. These methods are typically
calibrated to capture the entirety of the data, not just the signal. In this
paper, we demonstrate the importance of acknowledging noise when calibrating
hyperparameters and present a framework that enables users to do so. We use
this framework to explore the role hyperparameter calibration plays in
overfitting the data when applying t-SNE and UMAP. More specifically, we show
previously recommended values for perplexity and n_neighbors are too small and
overfit the noise. We also provide a workflow others may use to calibrate
hyperparameters in the presence of noise.
