  Context: Advancements in machine learning (ML) lead to a shift from the
traditional view of software development, where algorithms are hard-coded by
humans, to ML systems materialized through learning from data. Therefore, we
need to revisit our ways of developing software systems and consider the
particularities required by these new types of systems. Objective: The purpose
of this study is to systematically identify, analyze, summarize, and synthesize
the current state of software engineering (SE) research for engineering ML
systems. Method: I performed a systematic literature review (SLR). I
systematically selected a pool of 141 studies from SE venues and then conducted
a quantitative and qualitative analysis using the data extracted from these
studies. Results: The non-deterministic nature of ML systems complicates all SE
aspects of engineering ML systems. Despite increasing interest from 2018
onwards, the results reveal that none of the SE aspects have a mature set of
tools and techniques. Testing is by far the most popular area among
researchers. Even for testing ML systems, engineers have only some tool
prototypes and solution proposals with weak experimental proof. Many of the
challenges of ML systems engineering were identified through surveys and
interviews. Researchers should conduct experiments and case studies, ideally in
industrial environments, to further understand these challenges and propose
solutions. Conclusion: The results may benefit (1) practitioners in foreseeing
the challenges of ML systems engineering; (2) researchers and academicians in
identifying potential research questions; and (3) educators in designing or
updating SE courses to cover ML systems engineering.
