  Discriminative deep learning models with a linear+softmax final layer have a
problem: the latent space only predicts the conditional probabilities $p(Y|X)$
but not the full joint distribution $p(Y,X)$, which necessitates a generative
approach. The conditional probability cannot detect outliers, causing outlier
sensitivity in softmax networks. This exacerbates model over-confidence
impacting many problems, such as hallucinations, confounding biases, and
dependence on large datasets. To address this we introduce a novel embedding
constraint based on the Method of Moments (MoM). We investigate the use of
polynomial moments ranging from 1st through 4th order hyper-covariance
matrices. Furthermore, we use this embedding constraint to train an
Axis-Aligned Gaussian Mixture Model (AAGMM) final layer, which learns not only
the conditional, but also the joint distribution of the latent space. We apply
this method to the domain of semi-supervised image classification by extending
FlexMatch with our technique. We find our MoM constraint with the AAGMM layer
is able to match the reported FlexMatch accuracy, while also modeling the joint
distribution, thereby reducing outlier sensitivity. We also present a
preliminary outlier detection strategy based on Mahalanobis distance and
discuss future improvements to this strategy. Code is available at:
\url{https://github.com/mmajurski/ssl-gmm}
