  Fatal accidents are a major issue hindering the wide acceptance of
safety-critical systems that employ machine learning and deep learning models,
such as automated driving vehicles. In order to use machine learning in a
safety-critical system, it is necessary to demonstrate the safety and security
of the system through engineering processes. However, thus far, no such widely
accepted engineering concepts or frameworks have been established for these
systems. The key to using a machine learning model in a deductively engineered
system is decomposing the data-driven training of machine learning models into
requirement, design, and verification, particularly for machine learning models
used in safety-critical systems. Simultaneously, open problems and relevant
technical fields are not organized in a manner that enables researchers to
select a theme and work on it. In this study, we identify, classify, and
explore the open problems in engineering (safety-critical) machine learning
systems --- that is, in terms of requirement, design, and verification of
machine learning models and systems --- as well as discuss related works and
research directions, using automated driving vehicles as an example. Our
results show that machine learning models are characterized by a lack of
requirements specification, lack of design specification, lack of
interpretability, and lack of robustness. We also perform a gap analysis on a
conventional system quality standard SQuARE with the characteristics of machine
learning models to study quality models for machine learning systems. We find
that a lack of requirements specification and lack of robustness have the
greatest impact on conventional quality models.
