  We study all implementations of the $S_3$ symmetry in the two Higgs doublet
model with quarks, consistent with non-zero quark masses and a
Cabibbo-Kobayashi-Maskawa (CKM) matrix which is not block diagonal. We study
the impact of the various soft-breaking terms and vacuum expectation values,
and find an interesting relation between $\alpha$ and $\beta$. We also show
that, in this minimal setting, only two types of assignments are possible:
either all field sectors are in singlets or all field sectors have a doublet.
