  Statistics is sometimes described as the science of reasoning under
uncertainty. Statistical models provide one view of this uncertainty, but what
is frequently neglected is the invisible portion of uncertainty: that assumed
not to exist once a model has been fitted to some data. Systematic errors, i.e.
bias, in data relative to some model and inferential goal can seriously
undermine research conclusions, and qualitative and quantitative techniques
have been created across several disciplines to quantify and generally appraise
such potential biases. Perhaps best known are so-called risk of bias assessment
instruments used to investigate the likely quality of randomised controlled
trials in medical research. However, the logic of assessing the risks caused by
various types of systematic error to statistical arguments applies far more
widely. This logic applies even when statistical adjustment strategies for
potential biases are used, as these frequently make assumptions (e.g. data
missing at random) that can never be guaranteed in finite samples. Mounting
concern about such situations can be seen in the increasing calls for greater
consideration of biases caused by nonprobability sampling in descriptive
inference (i.e. survey sampling), and the statistical generalisability of
in-sample causal effect estimates in causal inference; both of which relate to
the consideration of model-based and wider uncertainty when presenting research
conclusions from models. Given that model-based adjustments are never perfect,
we argue that qualitative risk of bias reporting frameworks for both
descriptive and causal inferential arguments should be further developed and
made mandatory by journals and funders. It is only through clear statements of
the limits to statistical arguments that consumers of research can fully judge
their value for any specific application.
