  The emergence of quantization at the nanoscale, the quantum size effect
(QSE), allows flexible control of matter and is a rich source of advanced
functionalities. A QSE-induced transition into an insulating phase in
semimetallic nanofilms was predicted for bismuth a half-century ago and has
regained new interest with regard to its surface states exhibiting nontrivial
electronic topology. Here, we reveal an unexpected mechanism of the transition
by high-resolution angle-resolved photoelectron spectroscopy combined with
theoretical calculations. Anomalous evolution and degeneracy of quantized
energy levels indicate that increased Coulomb repulsion from the surface states
deforms a quantum confinement potential with decreasing thickness. The
potential deformation drastically modulates spatial distributions of quantized
wave functions, which leads to acceleration of the transition even beyond the
original QSE picture. This discovery establishes a complete picture of the
long-discussed problem in bismuth and highlights the new class of size effects
dominating nanoscale transport in systems with metallic surface states.
