  Online Just-In-Time Software Defect Prediction (O-JIT-SDP) uses an online
model to predict whether a new software change will introduce a bug or not.
However, existing studies neglect the interaction of Software Quality Assurance
(SQA) staff with the model, which may miss the opportunity to improve the
prediction accuracy through the feedback from SQA staff. To tackle this
problem, we propose Human-In-The-Loop (HITL) O-JIT-SDP that integrates feedback
from SQA staff to enhance the prediction process. Furthermore, we introduce a
performance evaluation framework that utilizes a k-fold distributed bootstrap
method along with the Wilcoxon signed-rank test. This framework facilitates
thorough pairwise comparisons of alternative classification algorithms using a
prequential evaluation approach. Our proposal enables continuous statistical
testing throughout the prequential process, empowering developers to make
real-time decisions based on robust statistical evidence. Through
experimentation across 10 GitHub projects, we demonstrate that our evaluation
framework enhances the credibility of model evaluation, and the incorporation
of HITL feedback elevates the prediction performance of online JIT-SDP models.
These advancements hold the potential to significantly enhance the value of
O-JIT-SDP for industrial applications.
