  Large language models (LLMs) have shown remarkable potential for problem
solving, with open source models achieving increasingly impressive performance
on benchmarks measuring areas from logical reasoning to mathematical ability.
Ensembling models can further improve capabilities across a variety of domains.
However, conventional methods of combining models at inference time such as
shallow fusion necessitate a shared vocabulary and tokenization, and
alternatives like fine-tuning for domain-specific performance are both time
consuming and computationally expensive. We therefore present an inference-time
ensembling algorithm aimed at "averaging" outputs from multiple LLMs and
illustrate its improved performance across multiple domains compared to its
constituent models alone. Character-wise ensemble decoding, CharED, finds the
marginal distribution of each character for an individual model and performs a
weighted average to generate an output, character by character. In coding,
math, and toxicity benchmarks, we find our proposed model able to combine
complimentary strengths of multiple LLMs, regardless of vocabulary,
tokenization, or model size.
