  We describe the symmetry group of the stabilizer polytope for any number $n$
of systems and any prime local dimension $d$. In the qubit case, the symmetry
group coincides with the linear and anti-linear Clifford operations. In the
case of qudits, the structure is somewhat richer: for $n=1$, it is a wreath
product of permutations of bases and permutations of the elements within each
basis. For $n>1$, the symmetries are given by affine symplectic similitudes.
These are the affine maps that preserve the symplectic form of the underlying
discrete phase space up to a non-zero multiplier. We phrase these results with
respect to a number of a priori different notions of "symmetry'', including
Kadison symmetries (bijections that are compatible with convex combinations),
Wigner symmetries (bijections that preserve inner products), and symmetries
realized by an action on Hilbert space. Going beyond stabilizer states, we
extend an observation of Heinrich and Gross (Ref. [25]) and show that the
symmetries of fairly general sets of Hermitian operators are constrained by
certain moments. In particular: the symmetries of a set that behaves like a
3-design preserve Jordan products and are therefore realized by conjugation
with unitaries or anti-unitaries. (The structure constants of the Jordan
algebra are encoded in an order-three tensor, which we connect to the third
moments of a design). This generalizes Kadison's formulation of the classic
Wigner Theorem on quantum mechanical symmetries.
