  The increasing reliance on applications with machine learning (ML) components
calls for mature engineering techniques that ensure these are built in a robust
and future-proof manner. We aim to empirically determine the state of the art
in how teams develop, deploy and maintain software with ML components. We mined
both academic and grey literature and identified 29 engineering best practices
for ML applications. We conducted a survey among 313 practitioners to determine
the degree of adoption for these practices and to validate their perceived
effects. Using the survey responses, we quantified practice adoption,
differentiated along demographic characteristics, such as geography or team
size. We also tested correlations and investigated linear and non-linear
relationships between practices and their perceived effect using various
statistical models. Our findings indicate, for example, that larger teams tend
to adopt more practices, and that traditional software engineering practices
tend to have lower adoption than ML specific practices. Also, the statistical
models can accurately predict perceived effects such as agility, software
quality and traceability, from the degree of adoption for specific sets of
practices. Combining practice adoption rates with practice importance, as
revealed by statistical models, we identify practices that are important but
have low adoption, as well as practices that are widely adopted but are less
important for the effects we studied. Overall, our survey and the analysis of
responses received provide a quantitative basis for assessment and step-wise
improvement of practice adoption by ML teams.
