  Human language is a unique form of communication in the natural world,
distinguished by its structured nature. Most fundamentally, it is systematic,
meaning that signals can be broken down into component parts that are
individually meaningful -- roughly, words -- which are combined in a regular
way to form sentences. Furthermore, the way in which these parts are combined
maintains a kind of locality: words are usually concatenated together, and they
form contiguous phrases, keeping related parts of sentences close to each
other. We address the challenge of understanding how these basic properties of
language arise from broader principles of efficient communication under
information processing constraints. Here we show that natural-language-like
systematicity arises from minimization of excess entropy, a measure of
statistical complexity that represents the minimum amount of information
necessary for predicting the future of a sequence based on its past. In
simulations, we show that codes that minimize excess entropy factorize their
source distributions into approximately independent components, and then
express those components systematically and locally. Next, in a series of
massively cross-linguistic corpus studies, we show that human languages are
structured to have low excess entropy at the level of phonology, morphology,
syntax, and semantics. Our result suggests that human language performs a
sequential generalization of Independent Components Analysis on the statistical
distribution over meanings that need to be expressed. It establishes a link
between the statistical and algebraic structure of human language, and
reinforces the idea that the structure of human language may have evolved to
minimize cognitive load while maximizing communicative expressiveness.
