  In order to process efficiently ever-higher dimensional data such as images,
sentences, or audio recordings, one needs to find a proper way to reduce the
dimensionality of such data. In this regard, SVD-based methods including PCA
and Isomap have been extensively used. Recently, a neural network alternative
called autoencoder has been proposed and is often preferred for its higher
flexibility. This work aims to show that PCA is still a relevant technique for
dimensionality reduction in the context of classification. To this purpose, we
evaluated the performance of PCA compared to Isomap, a deep autoencoder, and a
variational autoencoder. Experiments were conducted on three commonly used
image datasets: MNIST, Fashion-MNIST, and CIFAR-10. The four different
dimensionality reduction techniques were separately employed on each dataset to
project data into a low-dimensional space. Then a k-NN classifier was trained
on each projection with a cross-validated random search over the number of
neighbours. Interestingly, our experiments revealed that k-NN achieved
comparable accuracy on PCA and both autoencoders' projections provided a big
enough dimension. However, PCA computation time was two orders of magnitude
faster than its neural network counterparts.
