  We study the phase transition from a topological to a normal insulator with
concentration $x$ in (Bi$_{1-x}$In$_{x})_2$Se$_3$ and
(Bi$_{1-x}$Sb$_{x})_2$Se$_3$ in the Bi$_2$Se$_3$ crystal structure. We carry
out first-principles calculations on small supercells, using this information
to build Wannierized effective Hamiltonians for a more realistic treatment of
disorder. Despite the fact that the spin-orbit coupling (SOC) strength is
similar in In and Sb, we find that the critical concentration $x_{\rm c}$ is
much smaller in (Bi$_{1-x}$In$_{x})_2$Se$_3$ than in
(Bi$_{1-x}$Sb$_{x})_2$Se$_3$. For example, the direct supercell calculations
suggest that $x_{\rm c}$ is below 12.5% and above 87.5$%$ for the two alloys
respectively. More accurate results are obtained from realistic disordered
calculations, where the topological properties of the disordered systems are
understood from a statistical point of view. Based on these calculations, $x_c$
is around 17% for (Bi$_{1-x}$In$_{x})_2$Se$_3$, but as high as 78%-83% for
(Bi$_{1-x}$Sb$_{x})_2$Se$_3$. In (Bi$_{1-x}$Sb$_{x})_2$Se$_3$, we find that the
phase transition is dominated by the decrease of SOC, with a crossover or
"critical plateau" observed from around 78$%$ to 83$%$. On the other hand, for
(Bi$_{1-x}$In$_{x})_2$Se$_3$, the In 5$s$ orbitals suppress the topological
band inversion at low impurity concentration, therefore accelerating the phase
transition. In (Bi$_{1-x}$In$_{x})_2$Se$_3$ we also find a tendency of In atoms
to segregate.
