  Even if a probability distribution is properly normalizable, its associated
Shannon (or von Neumann) entropy can easily be infinite. We carefully analyze
conditions under which this phenomenon can occur. Roughly speaking, this
happens when arbitrarily small amounts of probability are dispersed into an
infinite number of states; we shall quantify this observation and make it
precise. We develop several particularly simple, elementary, and useful bounds,
and also provide some asymptotic estimates, leading to necessary and sufficient
conditions for the occurrence of infinite Shannon entropy. We go to some effort
to keep technical computations as simple and conceptually clear as possible. In
particular, we shall see that large entropies cannot be localized in state
space; large entropies can only be supported on an exponentially large number
of states. We are for the time being interested in single-channel Shannon
entropy in the information theoretic sense, not entropy in a stochastic field
theory or QFT defined over some configuration space, on the grounds that this
simple problem is a necessary precursor to understanding infinite entropy in a
field theoretic context.
