  The non-backtracking operator was recently shown to provide a significant
improvement when used for spectral clustering of sparse networks. In this paper
we analyze its spectral density on large random sparse graphs using a mapping
to the correlation functions of a certain interacting quantum disordered system
on the graph. On sparse, tree-like graphs, this can be solved efficiently by
the cavity method and a belief propagation algorithm. We show that there exists
a paramagnetic phase, leading to zero spectral density, that is stable outside
a circle of radius $\sqrt{\rho}$, where $\rho$ is the leading eigenvalue of the
non-backtracking operator. We observe a second-order phase transition at the
edge of this circle, between a zero and a non-zero spectral density. That fact
that this phase transition is absent in the spectral density of other matrices
commonly used for spectral clustering provides a physical justification of the
performances of the non-backtracking operator in spectral clustering.
