  We provide a formulation of Stochastic Inflation in full general relativity
that goes beyond the slow-roll and separate universe approximations. We show
how gauge invariant Langevin source terms can be obtained for the complete set
of Einstein equations in their ADM formulation by providing a recipe for
coarse-graining the spacetime in any small gauge. These stochastic source terms
are defined in terms of the only dynamical scalar degree of freedom in
single-field inflation and all depend simply on the first two time derivatives
of the coarse-graining window function, on the gauge-invariant mode functions
that satisfy the Mukhanov-Sasaki evolution equation, and on the slow-roll
parameters. It is shown that this reasoning can also be applied to include
gravitons as stochastic sources, thus enabling the study of all relevant
degrees of freedom of general relativity for inflation. We validate the
efficacy of these Langevin dynamics directly using an example in uniform field
gauge, obtaining the stochastic e-fold number in the long wavelength limit
without the need for a first-passage-time analysis. As well as investigating
the most commonly used gauges in cosmological perturbation theory, we also
derive stochastic source terms for the coarse-grained BSSN formulation of
Einstein's equations, which enables a well-posed implementation for 3+1
numerical relativity simulations.
