  The standard theoretical understanding of the lattice thermal conductivity,
$\kappa_{\ell}$, of semiconductor alloys assumes that mass disorder is the most
important source of phonon scattering. In contrast, we show that the hitherto
neglected contribution of force-constant (IFC) disorder is essential to
accurately predict the $\kappa_{\ell}$ of those polar compounds characterized
by a complex atomic-scale structure. We have developed an \emph{ab initio}
method based on special quasirandom structures and Green's functions, and
including the role of IFC disorder, and applied it in order to calculate the
$\kappa_{\ell}$ of $\mathrm{In_{1-x}Ga_xAs}$ and $\mathrm{Si_{1-x}Ge_x}$
alloys. We show that, while for $\mathrm{Si_{1-x}Ge_x}$, phonon-alloy
scattering is dominated by mass disorder, for $\mathrm{In_{1-x}Ga_xAs}$, the
inclusion of IFC disorder is fundamental to accurately reproduce the
experimentally observed $\kappa_{\ell}$. As the presence of a complex
atomic-scale structure is common to most III-V and II-VI random semiconductor
alloys, we expect our method to be suitable for a wide class of materials.
