  The Taishan Antineutrino Observatory (TAO) is a ton-level liquid scintillator
detector to be placed at 30\,m from a core of the Taishan Nuclear Power Plant
for precise reactor antineutrino spectrum measurements. One important
background for TAO physics are the interactions of ambient neutrons that can
penetrate its outer shieldings. The neutrons fluence and energy spectrum are
measured with a Bonner sphere spectrometer. Data is unfolded with the iterative
Maximum-Likelihood Expectation-Maximization (MLEM) method. A simulation based
on Geant4 is performed to provide the initial input spectrum to the unfolding
and to understand the unfolded result. The total neutron fluence rate is
measured to be 36.1 $\pm$ 4.7 $Hz/m^2$, which is higher than the expectation.
For neutrons with kinetic energy lower than 20\,MeV, the measured fluence rate
and energy spectrum can be well reproduced by simulation. While for the region
greater than 20\,MeV, a significant discrepancy is observed and shall be
understood with further studies.
