  Machine-learning systems such as self-driving cars or virtual assistants are
composed of a large number of machine-learning models that recognize image
content, transcribe speech, analyze natural language, infer preferences, rank
options, etc. Models in these systems are often developed and trained
independently, which raises an obvious concern: Can improving a
machine-learning model make the overall system worse? We answer this question
affirmatively by showing that improving a model can deteriorate the performance
of downstream models, even after those downstream models are retrained. Such
self-defeating improvements are the result of entanglement between the models
in the system. We perform an error decomposition of systems with multiple
machine-learning models, which sheds light on the types of errors that can lead
to self-defeating improvements. We also present the results of experiments
which show that self-defeating improvements emerge in a realistic stereo-based
detection system for cars and pedestrians.
