  We continue classification of finite groups which can be used as symmetry
group of the scalar sector of the four-Higgs-doublet model (4HDM). We
systematically construct non-abelian groups via the group extension procedure,
starting from the known abelian groups $A$ and their automorphism groups
$\mbox{Aut}(A)$. Previously, we considered all cyclic groups $A$ available for
the 4HDM scalar sector. Here, we further develop the method and apply it to
extensions by the remaining rephasing groups $A$, namely $A =
\mathbb{Z}_2\times\mathbb{Z}_2$, $\mathbb{Z}_4\times \mathbb{Z}_2$, and
$\mathbb{Z}_2\times \mathbb{Z}_2\times \mathbb{Z}_2$. As $\mbox{Aut}(A)$ grows,
the procedure becomes more laborious, but we prove an isomorphism theorem which
helps classify all the options. We also comment on what remains to be done to
complete the classification of all finite non-abelian groups realizable in the
4HDM scalar sector without accidental continuous symmetries.
