  Using spin- and angle-resolved spectroscopy and relativistic many-body
calculations, we investigate the evolution of the electronic structure of
(Bi$_{1-x}$In$_x$)$_2$Se$_3$ bulk single crystals around the critical point of
the trivial to topological insulator quantum-phase transition. By increasing
$x$, we observe how a surface gap opens at the Dirac point of the initially
gapless topological surface state of Bi$_2$Se$_3$, leading to the existence of
massive fermions. The surface gap monotonically increases for a wide range of
$x$ values across the topological and trivial sides of the quantum-phase
transition. By means of photon-energy dependent measurements, we demonstrate
that the gapped surface state survives the inversion of the bulk bands which
occurs at a critical point near $x=0.055$. The surface state exhibits a
non-zero in-plane spin polarization which decays exponentially with increasing
$x$, and that persists on both the topological and trivial insulator phases.
Its out-of-plane spin polarization remains zero demonstrating the absence of a
hedgehog spin texture expected from broken time-reversal symmetry. Our
calculations reveal qualitative agreement with the experimental results all
across the quantum-phase transition upon the systematic variation of the
spin-orbit coupling strength. A non-time reversal symmetry breaking mechanism
of bulk-mediated scattering processes that increase with decreasing spin-orbit
coupling strength is proposed as explanation.
