  We propose a unified solution with $Z_4$ discrete symmetry for small neutrino
masses and stability of dark matter. Our model consists of an inert doublet
scalar, a dark singlet scalar $S$, a spurion scalar $\varphi$ and right-handed
neutrinos beyond the Standard Model (SM), which transform nontrivially under
$Z_4$. Once the $Z_4$ symmetry is broken to $Z_2$ by the VEV of $\varphi$, much
below the mass scale of the singlet scalar $S$, a small quartic coupling
between SM Higgs and inert doublets, which violates both $Z_4$ and lepton
number, is generated at tree level, so small neutrino masses are obtained at
one-loops. We discuss the consistent parameter space for neutrino masses,
direct detection and thermal leptogenesis in the model.
