  There is an ongoing debate on whether neural networks can grasp the
quasi-regularities in languages like humans. In a typical quasi-regularity
task, English past tense inflections, the neural network model has long been
criticized that it learns only to generalize the most frequent pattern, but not
the regular pattern, thus can not learn the abstract categories of regular and
irregular and is dissimilar to human performance. In this work, we train a set
of transformer models with different settings to examine their behavior on this
task. The models achieved high accuracy on unseen regular verbs and some
accuracy on unseen irregular verbs. The models' performance on the regulars is
heavily affected by type frequency and ratio but not token frequency and ratio,
and vice versa for the irregulars. The different behaviors on the regulars and
irregulars suggest that the models have some degree of symbolic learning on the
regularity of the verbs. In addition, the models are weakly correlated with
human behavior on nonce verbs. Although the transformer model exhibits some
level of learning on the abstract category of verb regularity, its performance
does not fit human data well, suggesting that it might not be a good cognitive
model.
