  Ensuring high-quality data is paramount for maximizing the performance of
machine learning models and business intelligence systems. However, challenges
in data quality, including noise in data capture, missing records, limited data
production, and confounding variables, significantly constrain the potential
performance of these systems. In this study, we propose an
architecture-agnostic algorithm, Gradient Guided Hypotheses (GGH), designed to
address these challenges. GGH analyses gradients from hypotheses as a proxy of
distinct and possibly contradictory patterns in the data. This framework
entails an additional step in machine learning training, where gradients can be
included or excluded from backpropagation. In this manner, missing and noisy
data are addressed through a unified solution that perceives both challenges as
facets of the same overarching issue: the propagation of erroneous information.
Experimental validation of GGH is conducted using real-world open-source
datasets, where records with missing rates of up to 98.5% are simulated.
Comparative analysis with state-of-the-art imputation methods demonstrates a
substantial improvement in model performance achieved by GGH. Specifically in
very high scarcity regimes, GGH was found to be the only viable solution.
Additionally, GGH's noise detection capabilities are showcased by introducing
simulated noise into the datasets and observing enhanced model performance
after filtering out the noisy data. This study presents GGH as a promising
solution for improving data quality and model performance in various
applications.
