  Social media is daily creating massive multimedia content with paired image
and text, presenting the pressing need to automate the vision and language
understanding for various multimodal classification tasks. Compared to the
commonly researched visual-lingual data, social media posts tend to exhibit
more implicit image-text relations. To better glue the cross-modal semantics
therein, we capture hinting features from user comments, which are retrieved
via jointly leveraging visual and lingual similarity. Afterwards, the
classification tasks are explored via self-training in a teacher-student
framework, motivated by the usually limited labeled data scales in existing
benchmarks. Substantial experiments are conducted on four multimodal social
media benchmarks for image text relation classification, sarcasm detection,
sentiment classification, and hate speech detection. The results show that our
method further advances the performance of previous state-of-the-art models,
which do not employ comment modeling or self-training.
