  The in-plane $\rho_{ab}(H,T)$ and the out-of-plane $\rho_c(H,T)$
magneto-transport in magnetic fields up to 28 T has been investigated in high
quality non-superconducting (down to 20 mK) La-free
Bi$_{2+x}$Sr$_{2-x}$CuO$_{6+\delta}$ single crystal. By measuring the angular
dependence of the in-plane and out-of-plane magnetoresistivities at
temperatures from 1 K down to 30 mK, we present evidence for the presence of
vortex-like excitations in a non-superconducting cuprate in the insulating
state. Such excitations have previously been observed by the detection of a
Nernst signal in superconducting cuprates at $T>T_c$ in magnetic fields.
