  In this paper we consider the parity violating gravity model within the
framework of teleparallel gravity. The parity violations are caused by the
couplings of a scalar field to the scalar invariants which are parity-odd and
quadratic in the torsion tensor. Totally there are two such type independent
invariants, and one of them is the Nieh-Yan density. Through investigations on
the cosmological perturbations of this model, we find that in general it
suffers from the difficulties of ghost instability in the scalar and vector
perturbations. But in the special case only the coupling to the Nieh-Yan
density exists, this model is ghost free and reduces to the Nieh-Yan modified
Teleparallel Gravity model. We also analyze the severity of the ghost
instability by studying the perturbations around the Minkowski background.
