  Let $\Gamma_n(\mathcal{\scriptstyle{O}}_\mathbb{K})$ denote the Hermitian
modular group of degree $n$ over an imaginary-quadratic number field
$\mathbb{K}$. In this paper we determine its maximal discrete extension in
$SU(n,n;\mathbb{C})$, which coincides with the normalizer of
$\Gamma_n(\mathcal{\scriptstyle{O}}_{\mathbb{K}})$. The description involves
the $n$-torsion subgroup of the ideal class group of $\mathbb{K}$. This group
is defined over a particular number field $\widehat{\mathbb{K}}_n$ and we can
describe the ramified primes in it. In the case $n=2$ we give an explicit
description, which involves generalized Atkin-Lehner involutions. Moreover we
find a natural characterization of this group in $SO(2,4)$.
