  Moir\'e superlattices engineer band properties and enable observation of
fractal energy spectra of Hofstadter butterfly. Recently, correlated-electron
physics hosted by flat bands in small-angle moir\'e systems has been at the
foreground. However, the implications of moir\'e band topology within the
single-particle framework are little explored experimentally. An outstanding
problem is understanding the effect of band topology on Hofstadter physics,
which does not require electron correlations. Our work experimentally studies
Chern state switching in the Hofstadter regime using twisted double bilayer
graphene (TDBG), which offers electric field tunable topological bands, unlike
twisted bilayer graphene. Here we show that the nontrivial topology reflects in
the Hofstadter spectra, in particular, by displaying a cascade of Hofstadter
gaps that switch their Chern numbers sequentially while varying the
perpendicular electric field. Our experiments together with theoretical
calculations suggest a crucial role of charge polarization changing
concomitantly with topological transitions in this system. Layer polarization
is likely to play an important role in the topological states in few-layer
twisted systems. Moreover, our work establishes TDBG as a novel Hofstadter
platform with nontrivial magnetoelectric coupling.
