  We introduce a probabilistic framework for quantifying the semantic
similarity between two groups of embeddings. We formulate the task of semantic
similarity as a model comparison task in which we contrast a generative model
which jointly models two sentences versus one that does not. We illustrate how
this framework can be used for the Semantic Textual Similarity tasks using
clear assumptions about how the embeddings of words are generated. We apply
model comparison that utilises information criteria to address some of the
shortcomings of Bayesian model comparison, whilst still penalising model
complexity. We achieve competitive results by applying the proposed framework
with an appropriate choice of likelihood on the STS datasets.
