  Software engineering techniques are increasingly relying on deep learning
approaches to support many software engineering tasks, from bug triaging to
code generation. To assess the efficacy of such techniques researchers
typically perform controlled experiments. Conducting these experiments,
however, is particularly challenging given the complexity of the space of
variables involved, from specialized and intricate architectures and algorithms
to a large number of training hyper-parameters and choices of evolving
datasets, all compounded by how rapidly the machine learning technology is
advancing, and the inherent sources of randomness in the training process. In
this work we conduct a mapping study, examining 194 experiments with techniques
that rely on deep neural networks appearing in 55 papers published in premier
software engineering venues to provide a characterization of the
state-of-the-practice, pinpointing experiments common trends and pitfalls. Our
study reveals that most of the experiments, including those that have received
ACM artifact badges, have fundamental limitations that raise doubts about the
reliability of their findings. More specifically, we find: weak analyses to
determine that there is a true relationship between independent and dependent
variables (87% of the experiments); limited control over the space of DNN
relevant variables, which can render a relationship between dependent variables
and treatments that may not be causal but rather correlational (100% of the
experiments); and lack of specificity in terms of what are the DNN variables
and their values utilized in the experiments (86% of the experiments) to define
the treatments being applied, which makes it unclear whether the techniques
designed are the ones being assessed, or how the sources of extraneous
variation are controlled. We provide some practical recommendations to address
these limitations.
