  Estimating the entropy rate of discrete time series is a challenging problem
with important applications in numerous areas including neuroscience, genomics,
image processing and natural language processing. A number of approaches have
been developed for this task, typically based either on universal data
compression algorithms, or on statistical estimators of the underlying process
distribution. In this work, we propose a fully-Bayesian approach for entropy
estimation. Building on the recently introduced Bayesian Context Trees (BCT)
framework for modelling discrete time series as variable-memory Markov chains,
we show that it is possible to sample directly from the induced posterior on
the entropy rate. This can be used to estimate the entire posterior
distribution, providing much richer information than point estimates. We
develop theoretical results for the posterior distribution of the entropy rate,
including proofs of consistency and asymptotic normality. The practical utility
of the method is illustrated on both simulated and real-world data, where it is
found to outperform state-of-the-art alternatives.
