  Innovation and obsolescence describe dynamics of ever-churning and adapting
social and biological systems, concepts that encompass field-specific
formulations. We formalize the connection with a reduced model of the dynamics
of the "space of the possible" (e.g. technologies, mutations, theories) to
which agents (e.g. firms, organisms, scientists) couple as they grow, die, and
replicate. We predict three regimes: the space is finite, ever growing, or a
Schumpeterian dystopia in which obsolescence drives the system to collapse. We
reveal a critical boundary at which the space of the possible fluctuates
dramatically in size, displaying recurrent periods of minimal and of veritable
diversity. When the space is finite, corresponding to physically realizable
systems, we find surprising structure. This structure predicts a taxonomy for
the density of agents near and away from the innovative frontier that we
compare with distributions of firm productivity, covid diversity, and citation
rates for scientific publications. Remarkably, our minimal model derived from
first principles aligns with empirical examples, implying a follow-the-leader
dynamic in firm cost efficiency and biological evolution, whereas scientific
progress reflects consensus that waits on old ideas to go obsolete. Our theory
introduces a fresh and empirically testable framework for unifying innovation
and obsolescence across fields.
