  Rolling bearing fault diagnosis has garnered increased attention in recent
years owing to its presence in rotating machinery across various industries,
and an ever increasing demand for efficient operations. Prompt detection and
accurate prediction of bearing failures can help reduce the likelihood of
unexpected machine downtime and enhance maintenance schedules, averting lost
productivity. Recent technological advances have enabled monitoring the health
of these assets at scale using a variety of sensors, and predicting the
failures using modern Machine Learning (ML) approaches including deep learning
architectures. Vibration data has been collected using accelerated
run-to-failure of overloaded bearings, or by introducing known failure in
bearings, under a variety of operating conditions such as rotating speed, load
on the bearing, type of bearing fault, and data acquisition frequency. However,
in the development of bearing failure classification models using vibration
data there is a lack of consensus in the metrics used to evaluate the models,
data partitions used to evaluate models, and methods used to generate failure
labels in run-to-failure experiments. An understanding of the impact of these
choices is important to reliably develop models, and deploy them in practical
settings. In this work, we demonstrate the significance of these choices on the
performance of the models using publicly-available vibration datasets, and
suggest model development considerations for real world scenarios. Our
experimental findings demonstrate that assigning vibration data from a given
bearing across training and evaluation splits leads to over-optimistic
performance estimates, PCA-based approach is able to robustly generate labels
for failure classification in run-to-failure experiments, and $F$ scores are
more insightful to evaluate the models with unbalanced real-world failure data.
