  Ensuring that analyses performed on a dataset are representative of the
entire population is one of the central problems in statistics. Most classical
techniques assume that the dataset is independent of the analyst's query and
break down in the common setting where a dataset is reused for multiple,
adaptively chosen, queries. This problem of \emph{adaptive data analysis} was
formalized in the seminal works of Dwork et al. (STOC, 2015) and Hardt and
Ullman (FOCS, 2014).
  We identify a remarkably simple set of assumptions under which the queries
will continue to be representative even when chosen adaptively: The only
requirements are that each query takes as input a random subsample and outputs
few bits. This result shows that the noise inherent in subsampling is
sufficient to guarantee that query responses generalize. The simplicity of this
subsampling-based framework allows it to model a variety of real-world
scenarios not covered by prior work.
  In addition to its simplicity, we demonstrate the utility of this framework
by designing mechanisms for two foundational tasks, statistical queries and
median finding. In particular, our mechanism for answering the broadly
applicable class of statistical queries is both extremely simple and state of
the art in many parameter regimes.
