  Partly motivated by entropy-estimation problems in neuroscience, we present a
detailed and extensive comparison between some of the most popular and
effective entropy estimation methods used in practice: The plug-in method, four
different estimators based on the Lempel-Ziv (LZ) family of data compression
algorithms, an estimator based on the Context-Tree Weighting (CTW) method, and
the renewal entropy estimator.
  **Methodology. Three new entropy estimators are introduced. For two of the
four LZ-based estimators, a bootstrap procedure is described for evaluating
their standard error, and a practical rule of thumb is heuristically derived
for selecting the values of their parameters. ** Theory. We prove that, unlike
their earlier versions, the two new LZ-based estimators are consistent for
every finite-valued, stationary and ergodic process. An effective method is
derived for the accurate approximation of the entropy rate of a finite-state
HMM with known distribution. Heuristic calculations are presented and
approximate formulas are derived for evaluating the bias and the standard error
of each estimator. ** Simulation. All estimators are applied to a wide range of
data generated by numerous different processes with varying degrees of
dependence and memory. Some conclusions drawn from these experiments include:
(i) For all estimators considered, the main source of error is the bias. (ii)
The CTW method is repeatedly and consistently seen to provide the most accurate
results. (iii) The performance of the LZ-based estimators is often comparable
to that of the plug-in method. (iv) The main drawback of the plug-in method is
its computational inefficiency.
