  Machine learning contrasts with traditional software development in that the
oracle is the data, and the data is not always a correct representation of the
problem that machine learning tries to model. We present a survey of the oracle
issues found in machine learning and state-of-the-art solutions for dealing
with these issues. These include lines of research for differential testing,
metamorphic testing, and test coverage. We also review some recent improvements
to robustness during modeling that reduce the impact of oracle issues, as well
as tools and frameworks for assisting in testing and discovering issues
specific to the dataset.
