  Recent work has shown that the quasinormal mode spectrum of black holes is
unstable under small perturbations (of order $\epsilon$) of the radial
potential, while the early time-domain ringdown waveform is only marginally
affected. In this paper we provide further insight into the apparent tension
between the frequency-domain and the time-domain descriptions by analyzing the
scattering properties of the problem. In the frequency domain, we study
analytically the solutions corresponding to the perturbed potential. We show
that there are two qualitatively different classes of instabilities, and that
both Schwarzschild and Kerr black holes are affected by what we call a "Type
II" instability, i.e., an exponential migration of the mode frequencies away
from their unperturbed value as the perturbing "bump" moves away from the peak
of the unperturbed potential. In the time domain, we elucidate the effect of
the spectral instability in terms of the causal structure of the Green's
function. By using an equivalent scattering problem we confirm analytically
(and show numerically) that the deviation from the unperturbed waveform in the
early ringdown stage is proportional to $\epsilon$ when
$\epsilon\lesssim10^{-2}$.
