  This is a detailed tutorial paper which explains the Principal Component
Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start
with projection, PCA with eigen-decomposition, PCA with one and multiple
projection directions, properties of the projection matrix, reconstruction
error minimization, and we connect to autoencoder. Then, PCA with singular
value decomposition, dual PCA, and kernel PCA are covered. SPCA using both
scoring and Hilbert-Schmidt independence criterion are explained. Kernel SPCA
using both direct and dual approaches are then introduced. We cover all cases
of projection and reconstruction of training and out-of-sample data. Finally,
some simulations are provided on Frey and AT&T face datasets for verifying the
theory in practice.
