  We present our 7th place solution to the Gendered Pronoun Resolution
challenge, which uses BERT without fine-tuning and a novel augmentation
strategy designed for contextual embedding token-level tasks. Our method
anonymizes the referent by replacing candidate names with a set of common
placeholder names. Besides the usual benefits of effectively increasing
training data size, this approach diversifies idiosyncratic information
embedded in names. Using same set of common first names can also help the model
recognize names better, shorten token length, and remove gender and regional
biases associated with names. The system scored 0.1947 log loss in stage 2,
where the augmentation contributed to an improvements of 0.04. Post-competition
analysis shows that, when using different embedding layers, the system scores
0.1799 which would be third place.
