  Artificial Intelligence and Machine Learning have witnessed rapid,
significant improvements in Natural Language Processing (NLP) tasks. Utilizing
Deep Learning, researchers have taken advantage of repository comments in
Software Engineering to produce accurate methods for detecting Self-Admitted
Technical Debt (SATD) from 20 open-source Java projects' code. In this work, we
improve SATD detection with a novel approach that leverages the Bidirectional
Encoder Representations from Transformers (BERT) architecture. For comparison,
we re-evaluated previous deep learning methods and applied stratified 10-fold
cross-validation to report reliable F$_1$-scores. We examine our model in both
cross-project and intra-project contexts. For each context, we use re-sampling
and duplication as augmentation strategies to account for data imbalance. We
find that our trained BERT model improves over the best performance of all
previous methods in 19 of the 20 projects in cross-project scenarios. However,
the data augmentation techniques were not sufficient to overcome the lack of
data present in the intra-project scenarios, and existing methods still perform
better. Future research will look into ways to diversify SATD datasets in order
to maximize the latent power in large BERT models.
