  Any non-trivial scattering with any massless fields in four spacetime
dimensions will generically produce an "out" state with memory which gives rise
to infrared divergences in the standard $S$-matrix. To obtain an
infrared-finite scattering theory, one must suitably include states with
memory. However, except in the case of QED with massive charged particles,
asymptotic states with memory that have finite energy and angular momentum have
not been constructed for more general theories (e.g. massless QED, Yang-Mills
and quantum gravity). To this end, we construct direct-integral representations
over the "Lorentz orbit" of a given memory and classify all "orbit space
representations" that have well-defined energy and angular momentum. We thereby
provide an explicit construction of a large supply of physical states with
memory as well as the explicit action of the BMS charges all states. The
construction of such states is a key step toward the formulation of an
infrared-finite scattering theory. While we primarily focus on the quantum
gravitational case, we outline how the methods presented in this paper can be
applied to obtain representations of the Poincar\'e group with memory for more
general quantum field theories.
