  The incorporation of Wilson lines leads to an extension of the modular
symmetries of string compactification beyond $\mathrm{SL}(2,\mathbb Z)$. In the
simplest case with one Wilson line $Z$, K\"ahler modulus $T$ and complex
structure modulus $U$, we are led to the Siegel modular group
$\mathrm{Sp}(4,\mathbb Z)$. It includes $\mathrm{SL}(2,\mathbb
Z)_T\times\mathrm{SL}(2,\mathbb Z)_U$ as well as $\mathbb Z_2$ mirror symmetry,
which interchanges $T$ and $U$. Possible applications to flavor physics of the
Standard Model require the study of orbifolds of $\mathrm{Sp}(4,\mathbb Z)$ to
obtain chiral fermions. We identify the 13 possible orbifolds and determine
their modular flavor symmetries as subgroups of $\mathrm{Sp}(4,\mathbb Z)$.
Some cases correspond to symmetric orbifolds that extend previously discussed
cases of $\mathrm{SL}(2,\mathbb Z)$. Others are based on asymmetric orbifold
twists (including mirror symmetry) that do no longer allow for a simple
intuitive geometrical interpretation and require further study. Sometimes they
can be mapped back to symmetric orbifolds with quantized Wilson lines. The
symmetries of $\mathrm{Sp}(4,\mathbb Z)$ reveal exciting new aspects of modular
symmetries with promising applications to flavor model building.
