  We introduce the partially observable history process (POHP) formalism for
reinforcement learning. POHP centers around the actions and observations of a
single agent and abstracts away the presence of other players without reducing
them to stochastic processes. Our formalism provides a streamlined interface
for designing algorithms that defy categorization as exclusively single or
multi-agent, and for developing theory that applies across these domains. We
show how the POHP formalism unifies traditional models including the Markov
decision process, the Markov game, the extensive-form game, and their partially
observable extensions, without introducing burdensome technical machinery or
violating the philosophical underpinnings of reinforcement learning. We
illustrate the utility of our formalism by concisely exploring observable
sequential rationality, examining some theoretical properties of general
immediate regret minimization, and generalizing the extensive-form regret
minimization (EFR) algorithm.
