  Machine learning models trained on code and related artifacts offer valuable
support for software maintenance but suffer from interpretability issues due to
their complex internal variables. These concerns are particularly significant
in safety-critical applications where the models' decision-making processes
must be reliable. The specific features and representations learned by these
models remain unclear, adding to the hesitancy in adopting them widely. To
address these challenges, we introduce DeepCodeProbe, a probing approach that
examines the syntax and representation learning abilities of ML models designed
for software maintenance tasks. Our study applies DeepCodeProbe to
state-of-the-art models for code clone detection, code summarization, and
comment generation. Findings reveal that while small models capture abstract
syntactic representations, their ability to fully grasp programming language
syntax is limited. Increasing model capacity improves syntax learning but
introduces trade-offs such as increased training time and overfitting.
DeepCodeProbe also identifies specific code patterns the models learn from
their training data. Additionally, we provide best practices for training
models on code to enhance performance and interpretability, supported by an
open-source replication package for broader application of DeepCodeProbe in
interpreting other code-related models.
