  The Cancer Registry of Norway (CRN) is a public body responsible for
capturing and curating cancer patient data histories to provide a unified
access to research data and statistics for doctors, patients, and policymakers.
For this purpose, CRN develops and operates a complex, constantly-evolving, and
socio-technical software system. Recently, machine learning (ML) algorithms
have been introduced into this system to augment the manual decisions made by
humans with automated decision support from learned models. To ensure that the
system is correct and robust and cancer patients' data are properly handled and
do not violate privacy concerns, automated testing solutions are being
developed. In this paper, we share the challenges that we identified when
developing automated testing solutions at CRN. Such testing potentially impacts
the quality of cancer data for years to come, which is also used by the
system's stakeholders to make critical decisions. The challenges identified are
not specific to CRN but are also valid in the context of other healthcare
registries. We also provide some details on initial solutions that we are
investigating to solve the identified challenges.
