  We study three-point functions of operators on the $1/2$ BPS Wilson loop in
planar $\mathcal{N}=4$ super Yang-Mills theory. The operators we consider are
"defect changing operators", which change the scalar coupled to the Wilson
loop. We first perform the computation at two loops in general set-ups, and
then study a special scaling limit called the ladders limit, in which the
spectrum is known to be described by a quantum mechanics with the
SL(2,$\mathbb{R}$) symmetry. In this limit, we resum the Feynman diagrams using
the Schwinger-Dyson equation and determine the structure constants at all order
in the rescaled coupling constant. Besides providing an interesting solvable
example of defect conformal field theories, our result gives invaluable data
for the integrability-based approach to the structure constants.
