  Recent work has shown that neural feature- and representation-learning, e.g.
BERT, achieves superior performance over traditional manual feature engineering
based approaches, with e.g. SVMs, in translationese classification tasks.
Previous research did not show $(i)$ whether the difference is because of the
features, the classifiers or both, and $(ii)$ what the neural classifiers
actually learn. To address $(i)$, we carefully design experiments that swap
features between BERT- and SVM-based classifiers. We show that an SVM fed with
BERT representations performs at the level of the best BERT classifiers, while
BERT learning and using handcrafted features performs at the level of an SVM
using handcrafted features. This shows that the performance differences are due
to the features. To address $(ii)$ we use integrated gradients and find that
$(a)$ there is indication that information captured by hand-crafted features is
only a subset of what BERT learns, and $(b)$ part of BERT's top performance
results are due to BERT learning topic differences and spurious correlations
with translationese.
