  We explore the exactly duality invariant higher-derivative extension of
double field theory due to Hohm, Siegel and Zwiebach (HSZ) specialized to
cosmological backgrounds. Despite featuring a finite number of derivatives in
its original formulation, this theory encodes infinitely many $\alpha'$
corrections for metric, B-field and dilaton, which are obtained upon
integrating out certain extra fields. We perform a cosmological reduction with
fields depending only on time and show consistency of this truncation. We
compute the $\alpha'^4$ coefficients of the general cosmological
classification. As a possible model for how to deal with all $\alpha'$
corrections in string theory we give a two-derivative reformulation in which
the extra fields are kept. The corresponding Friedmann equations are then
ordinary second order differential equations that capture all $\alpha'$
corrections. We explore the tensionless limit $\alpha'\rightarrow \infty$,
which features string frame de Sitter vacua, and we set up perturbation theory
in $\frac{1}{\alpha'}$.
