  Detailed study of the solar magnetic field is crucial to understand its
generation, transport and reversals. The timing of the reversals may have
implications on space weather and thus identification of the temporal behavior
of the critical surges that lead to the polar field reversals is important. We
analyze the evolution of solar activity and magnetic flux transport in Cycles
21--24. We identify critical surges of remnant flux that reach the Sun's poles
and lead to the polar field reversals. We reexamine the polar field buildup and
reversals in their causal relation to the Sun's low-latitude activity. We
further identify the major remnant flux surges and their sources in the
time-latitude aspect. We find that special characteristics of individual
11-year cycles are generally determined by the spatiotemporal organization of
emergent magnetic flux and its unusual properties. We find a complicated
restructuring of high-latitude magnetic fields in Cycle~21. The global
rearrangements of solar magnetic fields were caused by surges of trailing and
leading polarities that occurred near the activity maximum. The decay of
non-Joy and anti-Hale active regions resulted in the remnant flux surges that
disturbed the usual order in magnetic flux transport. We finally show that the
leading-polarity surges during cycle minima sometimes link the following cycle
and a collective effect of these surges may lead to secular changes in solar
activity. The magnetic field from a Babcock--Leighton dynamo model generally
agrees with these observations.
