  If $t\rightarrow hq$ ($q=c,u$) or $h\rightarrow\tau\ell$ ($\ell=\mu,e$)
decays are observed, it will be a clear signal of new physics. We investigate
whether natural and viable flavor models can saturate the present direct upper
bounds without violating the indirect constraints from low energy loop
processes. We carry out our analysis in two theoretical frameworks: minimal
flavor violation (MFV) and Froggatt-Nielsen symmetry (FN). The simplest models
in either framework predict flavor changing couplings that are too small to be
directly observed. Yet, in the MFV framework, it is possible to have lepton
flavor changing Higgs couplings close to the bound if spurions related to heavy
singlet neutrinos play a role. In the FN framework, it is possible to have
large flavor changing couplings in both the up and the charged lepton sectors
if supersymmetry plays a role.
