  Hateful Meme Challenge proposed by Facebook AI has attracted contestants
around the world. The challenge focuses on detecting hateful speech in
multimodal memes. Various state-of-the-art deep learning models have been
applied to this problem and the performance on challenge's leaderboard has also
been constantly improved. In this paper, we enhance the hateful detection
framework, including utilizing Detectron for feature extraction, exploring
different setups of VisualBERT and UNITER models with different loss functions,
researching the association between the hateful memes and the sensitive text
features, and finally building ensemble method to boost model performance. The
AUROC of our fine-tuned VisualBERT, UNITER, and ensemble method achieves 0.765,
0.790, and 0.803 on the challenge's test set, respectively, which beats the
baseline models. Our code is available at
https://github.com/yatingtian/hateful-meme
