  Point-of-interest (POI) type prediction is the task of inferring the type of
a place from where a social media post was shared. Inferring a POI's type is
useful for studies in computational social science including sociolinguistics,
geosemiotics, and cultural geography, and has applications in geosocial
networking technologies such as recommendation and visualization systems. Prior
efforts in POI type prediction focus solely on text, without taking visual
information into account. However in reality, the variety of modalities, as
well as their semiotic relationships with one another, shape communication and
interactions in social media. This paper presents a study on POI type
prediction using multimodal information from text and images available at
posting time. For that purpose, we enrich a currently available data set for
POI type prediction with the images that accompany the text messages. Our
proposed method extracts relevant information from each modality to effectively
capture interactions between text and image achieving a macro F1 of 47.21
across eight categories significantly outperforming the state-of-the-art method
for POI type prediction based on text-only methods. Finally, we provide a
detailed analysis to shed light on cross-modal interactions and the limitations
of our best performing model.
