  Reproducibility is a cornerstone of scientific research, enabling validation,
extension, and progress. However, the rapidly evolving nature of software and
dependencies poses significant challenges to reproducing research results,
particularly in fields like adversarial robustness for deep neural networks,
where complex codebases and specialized toolkits are utilized. This paper
presents a case study of attempting to validate the results on certified
adversarial robustness in "SoK: Certified Robustness for Deep Neural Networks"
using the VeriGauge toolkit. Despite following the documented methodology,
numerous software and hardware compatibility issues were encountered, including
outdated or unavailable dependencies, version conflicts, and driver
incompatibilities. While a subset of the original results could be run, key
findings related to the empirical robust accuracy of various verification
methods proved elusive due to these technical obstacles, as well as slight
discrepancies in the test results. This practical experience sheds light on the
reproducibility crisis afflicting adversarial robustness research, where a lack
of reproducibility threatens scientific integrity and hinders progress. The
paper discusses the broader implications of this crisis, proposing potential
solutions such as containerization, software preservation, and comprehensive
documentation practices. Furthermore, it highlights the need for collaboration
and standardization efforts within the research community to develop robust
frameworks for reproducible research. By addressing the reproducibility crisis
head-on, this work aims to contribute to the ongoing discourse on scientific
reproducibility and advocate for best practices that ensure the reliability and
validity of research findings within not only adversarial robustness, but
security and technology research as a whole.
