  Bandit algorithms are increasingly used in real-world sequential
decision-making problems. Associated with this is an increased desire to be
able to use the resulting datasets to answer scientific questions like: Did one
type of ad lead to more purchases? In which contexts is a mobile health
intervention effective? However, classical statistical approaches fail to
provide valid confidence intervals when used with data collected with bandit
algorithms. Alternative methods have recently been developed for simple models
(e.g., comparison of means). Yet there is a lack of general methods for
conducting statistical inference using more complex models on data collected
with (contextual) bandit algorithms; for example, current methods cannot be
used for valid inference on parameters in a logistic regression model for a
binary reward. In this work, we develop theory justifying the use of
M-estimators -- which includes estimators based on empirical risk minimization
as well as maximum likelihood -- on data collected with adaptive algorithms,
including (contextual) bandit algorithms. Specifically, we show that
M-estimators, modified with particular adaptive weights, can be used to
construct asymptotically valid confidence regions for a variety of inferential
targets.
