  Adverse Event (ADE) extraction is one of the core tasks in digital
pharmacovigilance, especially when applied to informal texts. This task has
been addressed by the Natural Language Processing community using large
pre-trained language models, such as BERT. Despite the great number of
Transformer-based architectures used in the literature, it is unclear which of
them has better performances and why. Therefore, in this paper we perform an
extensive evaluation and analysis of 19 Transformer-based models for ADE
extraction on informal texts. We compare the performance of all the considered
models on two datasets with increasing levels of informality (forums posts and
tweets). We also combine the purely Transformer-based models with two
commonly-used additional processing layers (CRF and LSTM), and analyze their
effect on the models performance. Furthermore, we use a well-established
feature importance technique (SHAP) to correlate the performance of the models
with a set of features that describe them: model category (AutoEncoding,
AutoRegressive, Text-to-Text), pretraining domain, training from scratch, and
model size in number of parameters. At the end of our analyses, we identify a
list of take-home messages that can be derived from the experimental data.
