  Text classification is a fundamental language task in Natural Language
Processing. A variety of sequential models is capable making good predictions
yet there is lack of connection between language semantics and prediction
results. This paper proposes a novel influence score (I-score), a greedy search
algorithm called Backward Dropping Algorithm (BDA), and a novel feature
engineering technique called the "dagger technique". First, the paper proposes
a novel influence score (I-score) to detect and search for the important
language semantics in text document that are useful for making good prediction
in text classification tasks. Next, a greedy search algorithm called the
Backward Dropping Algorithm is proposed to handle long-term dependencies in the
dataset. Moreover, the paper proposes a novel engineering technique called the
"dagger technique" that fully preserve the relationship between explanatory
variable and response variable. The proposed techniques can be further
generalized into any feed-forward Artificial Neural Networks (ANNs) and
Convolutional Neural Networks (CNNs), and any neural network. A real-world
application on the Internet Movie Database (IMDB) is used and the proposed
methods are applied to improve prediction performance with an 81% error
reduction comparing with other popular peers if I-score and "dagger technique"
are not implemented.
