  We show that for any connected smooth manifold $M$ of dimension different
from $3$ the restriction of the compact-open topology to the diffeomorphism
group of $M$ is minimal, i.e. the group does not admit a strictly coarser
Hausdorff group topology. This implies the minimality of the compact-open
topology on the homeomorphism group of $M$ in all dimensions different from $3$
and $4$. In those cases for which in addition to all of this automatic
continuity is known to hold, such as when $M$ is closed, one can conclude that
the compact-open topology is the unique separable Hausdorff group topology on
the homeomorphism group.
