  Complex data pipelines are increasingly common in diverse applications such
as BI reporting and ML modeling. These pipelines often recur regularly (e.g.,
daily or weekly), as BI reports need to be refreshed, and ML models need to be
retrained. However, it is widely reported that in complex production pipelines,
upstream data feeds can change in unexpected ways, causing downstream
applications to break silently that are expensive to resolve.
  Data validation has thus become an important topic, as evidenced by notable
recent efforts from Google and Amazon, where the objective is to catch data
quality issues early as they arise in the pipelines. Our experience on
production data suggests, however, that on string-valued data, these existing
approaches yield high false-positive rates and frequently require human
intervention. In this work, we develop a corpus-driven approach to
auto-validate \emph{machine-generated data} by inferring suitable
data-validation "patterns" that accurately describe the underlying data domain,
which minimizes false positives while maximizing data quality issues caught.
Evaluations using production data from real data lakes suggest that
Auto-Validate is substantially more effective than existing methods. Part of
this technology ships as an Auto-Tag feature in Microsoft Azure Purview.
