  Transformer based pre-trained models such as BERT and its variants, which are
trained on large corpora, have demonstrated tremendous success for natural
language processing (NLP) tasks. Most of academic works are based on the
English language; however, the number of multilingual and language specific
studies increase steadily. Furthermore, several studies claimed that language
specific models outperform multilingual models in various tasks. Therefore, the
community tends to train or fine-tune the models for the language of their case
study, specifically. In this paper, we focus on Turkish maps data and
thoroughly evaluate both multilingual and Turkish based BERT, DistilBERT,
ELECTRA and RoBERTa. Besides, we also propose a MultiLayer Perceptron (MLP) for
fine-tuning BERT in addition to the standard approach of one-layer fine-tuning.
For the dataset, a mid-sized Address Parsing corpus taken with a relatively
high quality is constructed. Conducted experiments on this dataset indicate
that Turkish language specific models with MLP fine-tuning yields slightly
better results when compared to the multilingual fine-tuned models. Moreover,
visualization of address tokens' representations further indicates the
effectiveness of BERT variants for classifying a variety of addresses.
