  Viable neutrino and charged lepton masses and mixings are obtained by
imposing a $S_3 \times Z_4 \times Z_3$ flavor symmetry in a model with a few
additional Higgs. We use two $SU(2)_L$ triplet Higgs which are arranged as a
doublet of $S_3$, and standard model singlet Higgs which are also put as
doublets of $S_3$. We break the $S_3$ symmetry in this minimal model by giving
vacuum expectation values (VEV) to the additional Higgs fields. Dictated by the
minimum condition for the scalar potential, we obtain certain VEV alignments
which allow us to maintain $\mu-\tau$ symmetry in the neutrino sector, while
breaking it maximally for the charged leptons. This helps us to simultaneously
explain the hierarchical charged lepton masses, and the neutrino masses and
mixings. In particular, we obtain maximal $\theta_{23}$ and zero $\theta_{13}$.
We allow for a mild breaking of the $\mu-\tau$ symmetry for the neutrinos and
study the phenomenology. We give predictions for $\theta_{13}$ and the CP
violating Jarlskog invariant $J_{CP}$, as a function of the $\mu-\tau$ symmetry
breaking parameter. We also discuss possible collider signatures and
phenomenology associated with lepton flavor violating processes.
