  It is an essential product requirement of Yahoo Mail to distinguish between
personal and machine-generated emails. The old production classifier in Yahoo
Mail was based on a simple logistic regression model. That model was trained by
aggregating features at the SMTP address level. We propose building deep
learning models at the message level. We built and trained four individual CNN
models: (1) a content model with subject and content as input; (2) a sender
model with sender email address and name as input; (3) an action model by
analyzing email recipients' action patterns and correspondingly generating
target labels based on senders' opening/deleting behaviors; (4) a salutation
model by utilizing senders' "explicit salutation" signal as positive labels.
Next, we built a final full model after exploring different combinations of the
above four models. Experimental results on editorial data show that our full
model improves the adjusted-recall from 70.5% to 78.8% compared to the old
production model, while at the same time lifts the precision from 94.7% to
96.0%. Our full model also significantly beats the state-of-the-art Bert model
at this task. This full model has been deployed into the current production
system (Yahoo Mail 6).
