  Image memes and specifically their widely-known variation image macros, is a
special new media type that combines text with images and is used in social
media to playfully or subtly express humour, irony, sarcasm and even hate. It
is important to accurately retrieve image memes from social media to better
capture the cultural and social aspects of online phenomena and detect
potential issues (hate-speech, disinformation). Essentially, the background
image of an image macro is a regular image easily recognized as such by humans
but cumbersome for the machine to do so due to feature map similarity with the
complete image macro. Hence, accumulating suitable feature maps in such cases
can lead to deep understanding of the notion of image memes. To this end, we
propose a methodology, called Visual Part Utilization, that utilizes the visual
part of image memes as instances of the regular image class and the initial
image memes as instances of the image meme class to force the model to
concentrate on the critical parts that characterize an image meme.
Additionally, we employ a trainable attention mechanism on top of a standard
ViT architecture to enhance the model's ability to focus on these critical
parts and make the predictions interpretable. Several training and test
scenarios involving web-scraped regular images of controlled text presence are
considered for evaluating the model in terms of robustness and accuracy. The
findings indicate that light visual part utilization combined with sufficient
text presence during training provides the best and most robust model,
surpassing state of the art. Source code and dataset are available at
https://github.com/mever-team/memetector.
