  Qudit, a high-dimensional quantum system, provides a larger Hilbert space to
process the quantum information and has shown remarkable advantages over the
qubit counterparts. It is a great challenge to realize the high fidelity
universal quantum gates with qudits. Here we theoretically propose and
experimentally demonstrate a set of universal quantum gates for a single
optical qudit with four dimensions (including the generalized Pauli $X_4$ gate,
Pauli $Z_4$ gate, and all of their integer powers), which are encoded in the
polarization-spatial degree of freedom without multiple unstable cascaded
interferometers. Furthermore, we also realize the controlled-$X_4$ gate and all
of its integer powers. We have achieved both the ultra-high average gate
fidelity $99.73\%$ and efficiency $99.47\%$, which are above the the error
threshold for fault-tolerant quantum computation. Our work paves a way for the
large-scale high-dimensional fault-tolerant quantum computation with a
polynomial resource cost.
