  We investigate the recently developed Bidirectional Encoder Representations
from Transformers (BERT) model for the hyperpartisan news detection task. Using
a subset of hand-labeled articles from SemEval as a validation set, we test the
performance of different parameters for BERT models. We find that accuracy from
two different BERT models using different proportions of the articles is
consistently high, with our best-performing model on the validation set
achieving 85% accuracy and the best-performing model on the test set achieving
77%. We further determined that our model exhibits strong consistency, labeling
independent slices of the same article identically. Finally, we find that
randomizing the order of word pieces dramatically reduces validation accuracy
(to approximately 60%), but that shuffling groups of four or more word pieces
maintains an accuracy of about 80%, indicating the model mainly gains value
from local context.
