  LLM-as-a-judge approaches are a practical and effective way of assessing a
range of text tasks, aligning with human judgements especially when applied in
a comparative assessment fashion. However, when using pairwise comparisons to
rank a set of candidates the computational costs scale quadratically with the
number of candidates, which can have practical limitations. This paper
introduces a Product of Expert (PoE) framework for efficient LLM Comparative
Assessment. Here individual comparisons are considered experts that provide
information on a pair's score difference. The PoE framework combines the
information from these experts to yield an expression that can be maximized
with respect to the underlying set of candidates, and is highly flexible where
any form of expert can be assumed. When Gaussian experts are used one can
derive simple closed-form solutions for the optimal candidate ranking, as well
as expressions for selecting which comparisons should be made to maximize the
probability of this ranking. Our approach enables efficient comparative
assessment, where by using only a small subset of the possible comparisons, one
can generate score predictions that correlate as well to human judgements as
the predictions when all comparisons are used. We evaluate the approach on
multiple NLG tasks and demonstrate that our framework can yield considerable
computational savings when performing pairwise comparative assessment. When N
is large, with as few as 2% of comparisons the PoE solution can achieve similar
performance to when all comparisons are used.
