  Ambient neutrons may cause significant background for underground
experiments. Therefore, it is necessary to investigate their flux and energy
spectrum in order to devise a proper shielding. Here, two sets of altogether
ten moderated $^3$He neutron counters are used for a detailed study of the
ambient neutron background in tunnel IV of the Felsenkeller facility,
underground below 45 meters of rock in Dresden/Germany. One of the moderators
is lined with lead and thus sensitive to neutrons of energies higher than 10
MeV. For each $^3$He counter-moderator assembly, the energy dependent neutron
sensitivity was calculated with the FLUKA code. The count rates of the ten
detectors were then fitted with the MAXED and GRAVEL packages. As a result,
both the neutron energy spectrum from 10$^{-9}$ MeV to 300 MeV and the flux
integrated over the same energy range were determined experimentally.
  The data show that at a given depth, both the flux and the spectrum vary
significantly depending on local conditions. Energy integrated fluxes of $(0.61
\pm 0.05)$, $(1.96 \pm 0.15)$, and $(4.6 \pm 0.4) \times 10^{-4}$ cm$^{-2}$
s$^{-1}$, respectively, are measured for three sites within Felsenkeller tunnel
IV which have similar muon flux but different shielding wall configurations.
  The integrated neutron flux data and the obtained spectra for the three sites
are matched reasonably well by FLUKA Monte Carlo calculations that are based on
the known muon flux and composition of the measurement room walls.
