  A fundamental question for companies with large amount of logged data is: How
to use such logged data together with incoming streaming data to make good
decisions? Many companies currently make decisions via online A/B tests, but
wrong decisions during testing hurt users' experiences and cause irreversible
damage. A typical alternative is offline causal inference, which analyzes
logged data alone to make decisions. However, these decisions are not adaptive
to the new incoming data, and so a wrong decision will continuously hurt users'
experiences. To overcome the aforementioned limitations, we propose a framework
to unify offline causal inference algorithms (e.g., weighting, matching) and
online learning algorithms (e.g., UCB, LinUCB). We propose novel algorithms and
derive bounds on the decision accuracy via the notion of "regret". We derive
the first upper regret bound for forest-based online bandit algorithms.
Experiments on two real datasets show that our algorithms outperform other
algorithms that use only logged data or online feedbacks, or algorithms that do
not use the data properly.
