  We present spectroscopic observations for 64 radio galaxies having ultra
steep radio spectra. Twenty-nine objects have redshifts $z>2$, the largest
redshifts being almost 4. Our ultra steep spectrum (USS) criterion ($\alpha <
-1$) has proven to be the most efficient way of finding distant radio galaxies.
We find that even among the USS sources, there is a strong statistical
correlation between the spectral index and redshift. The most distant radio
galaxies within the USS sample have the steepest radio spectra.
  In our sample there are 3 radio galaxies at $z>3$ compared with 26 at $2 < z
< 3$. However, the present data do not allow us to decide whether there is a
decrease in co-moving source density at the highest redshifts.
  We have analyzed the spectra of the 30 objects with the highest redshifts
($z>1.9$). For these high redshift radio galaxies, Ly$\alpha$ is almost always
the dominant emission line, with a rest frame equivalent width ranging from
$\sim100$ \AA\ to more than 1000 \AA. The equivalent widths of the most
important emission lines (Ly$\alpha$, C\,{\small IV}, He\,{\small II},
C\,{\small III}]) are found to correlate strongly with each other. The large
rest frame equivalent widths and the correlation between the equivalent widths
of the emission lines, confirm that photoionization by a central continuum
source is most likely the dominant ionization mechanism.
  There are significant velocity differences between the various emission lines
of our high redshift radio galaxies; in particular the Ly$\alpha$ line is
shifted with respect to the higher ionization lines. Velocity shifts range from
100 to almost 1000 km s$^{-1}$ in some cases. Simulations show that the effects
of associated H\,{\small I} absorption on the Ly$\alpha$ emission line may be
