  In this perspective, we argue that despite the democratization of powerful
tools for data science and machine learning over the last decade, developing
the code for a trustworthy and effective data science system (DSS) is getting
harder. Perverse incentives and a lack of widespread software engineering (SE)
skills are among many root causes we identify that naturally give rise to the
current systemic crisis in reproducibility of DSSs. We analyze why SE and
building large complex systems is, in general, hard. Based on these insights,
we identify how SE addresses those difficulties and how we can apply and
generalize SE methods to construct DSSs that are fit for purpose. We advocate
two key development philosophies, namely that one should incrementally grow --
not biphasically plan and build -- DSSs, and one should always employ two types
of feedback loops during development: one which tests the code's correctness
and another that evaluates the code's efficacy.
