I have a 3D dataset with at least millions of data points (scatter events from atoms, approximately Gaussian). I am modeling this data with a Gaussian Mixture Model. The usual approach would be to optimize the GMM with the EM algorithm. However, I have substantial priors (e.g., the data is produced by a polymer made of light atoms) so the only way to find the correct GMM is via sampling.

Here's the problem: it's very expensive to compare a GMM to the data $D$, and I have to do that evaluation frequently. I would like to introduce an intermediate step: first model $G_d$ without priors, and then introduce priors and re-model $G_m$ where both are GMMs. Then while sampling $G_m$ I could just compare it to $G_d$ instead of $D$, which is a lot cheaper. So $G_d$ is kind of a data reduction. Intuitively the likelihood function for $G_m$ should be:

$$
p(D|G_m)\approx p(D|G_d)S(G_d|G_m)
$$

where $S(G_d|G_m)$ is some kind of scoring function (maybe K-L divergence? correlation?).
My questions are:


Can you re-model in this way and still get a good approximation of the likelihood?
What types of functions can be used for $S(G_d|G_m)$ and still get a proper PDF?
Does the choice of number of components in $G_d$ affect the final likelihood?


Below I attempt to illustrate this problem. The magenta $G_d$ is computed at the beginning. Then the red $G_m$ with its additional priors (including size and linearity) are fit to the magenta one, much faster than comparing to the data itself.

Quick clarification: the number of Gaussians in $G_m$ is much higher than $G_d$. That's because the only prior on $G_d$ is the Dirichlet one (so it's just regular Bayesian GMM fitting) whereas $G_m$ has much more detailed priors which enables us to make a more detailed model than the data itself would indicate.

Most classification models in fact don't yield a binary decision, but rather a continuous decision value (for instance, logistic regression models output a probability, SVMs output a signed distance to the hyperplane, ...). Using the decision values we can rank test samples, from 'almost certainly positive' to 'almost certainly negative'.

Based on the decision value, you can always assign some cutoff that configures the classifier in such a way that a certain fraction of data is labeled as positive. Determining an appropriate threshold can be done via the model's ROC or PR curves. You can play with the decision threshold regardless of the balance used in the training set. In other words, techniques like up -or downsampling are orthogonal to this.

Assuming the model is better than random, you can intuitively see that increasing the threshold for positive classification (which leads to less positive predictions) increases the model's precision at the cost of lower recall and vice versa.

Consider SVM as an intuitive example: the main challenge is to learn the orientation of the separating hyperplane. Up -or downsampling can help with this (I recommend preferring upsampling over downsampling). When the orientation of the hyperplane is good, we can play with the decision threshold (e.g. signed distance to the hyperplane) to get a desired fraction of positive predictions.
