  Let $\mathcal{M}_{n,d}$ be the moduli space of semi-stable rank $n$,
trace-free Higgs bundles with fixed determinant of degree $d$ on a Riemann
surface of genus at least $3$. We determine the following automorphism groups
of $\mathcal{M}_{n,d}$: (i) the group of automorphisms as a complex analytic
variety, (ii) the group of holomorphic symplectomorphisms, (iii) the group of
K\"ahler isomorphisms, (iv) the group of automorphisms of the quaternionic
structure, (v) the group of hyper-K\"ahler isomorphisms. When $n$ and $d$ are
coprime we show that $\mathcal{M}_{n,d}$ admits an anti-holomorphic isomorphism
if and only if the corresponding Riemann surface admits such a map. We then use
this to determine the isometry group of $\mathcal{M}_{n,d}$.
