  Obtaining meaningful quantitative descriptions of the statistical dependence
within multivariate systems is a difficult open problem. Recently, the Partial
Information Decomposition (PID) was proposed to decompose mutual information
(MI) about a target variable into components which are redundant, unique and
synergistic within different subsets of predictor variables. Here, we propose
to apply the elegant formalism of the PID to multivariate entropy, resulting in
a Partial Entropy Decomposition (PED). We implement the PED with an entropy
redundancy measure based on pointwise common surprisal; a natural definition
which is closely related to the definition of MI. We show how this approach can
reveal the dyadic vs triadic generative structure of multivariate systems that
are indistinguishable with classical Shannon measures. The entropy perspective
also shows that misinformation is synergistic entropy and hence that MI itself
includes both redundant and synergistic effects. We show the relationships
between the PED and MI in two predictors, and derive two alternative
information decompositions which we illustrate on several example systems. This
reveals that in entropy terms, univariate predictor MI is not a proper subset
of the joint MI, and we suggest this previously unrecognised fact explains in
part why obtaining a consistent PID has proven difficult. The PED also allows
separate quantification of mechanistic redundancy (related to the function of
the system) versus source redundancy (arising from dependencies between
inputs); an important distinction which no existing methods can address. The
new perspective provided by the PED helps to clarify some of the difficulties
encountered with the PID approach and the resulting decompositions provide
useful tools for practical data analysis across a wide range of application
areas.
