  Scientific computing libraries, either being in-house or open-source, have
experienced enormous progress in both engineering and scientific research. It
is therefore essential to ensure that the modifications in the source code
aroused by bug fixing or new feature development wouldn't compromise the
accuracy and functionality that has already been validated and verified. With
this in mind, this paper introduces a method for developing and implementing an
automatic regression test environment and takes the open-source multi-physics
library SPHinXsys \cite{zhang2021sphinxsys} as an example. Firstly, the
reference database for each benchmark test is generated from monitored data by
multiple executions. This database contains the maximum variation range of
metrics for different types of strategies, i.e., time-averaged method,
ensemble-averaged method as well as the dynamic time warping method, covering
the uncertainty arising from parallel computing, particle relaxation, physical
instabilities, etc. Then, new results obtained after source code modification
will be tested with them according to a curve-similarity based comparison.
Whenever the source code is updated, the regression test will be carried out
automatically for all test cases and used to report the validity of the current
results. This regression test environment has already been implemented in all
dynamics test cases released in SPHinXsys, including fluid dynamics, solid
mechanics, fluid-structure interaction, thermal and mass diffusion,
reaction-diffusion, and their multi-physics coupling, and shows good capability
for testing various problems. It's worth noting that while the present test
environment is built and implemented for a specific scientific computing
library, its underlying principle is generic and can be applied to many others.
