  Unlike traditional supervised learning, in many settings only partial
feedback is available. We may only observe outcomes for the chosen actions, but
not the counterfactual outcomes associated with other alternatives. Such
settings encompass a wide variety of applications including pricing, online
marketing and precision medicine. A key challenge is that observational data
are influenced by historical policies deployed in the system, yielding a biased
data distribution. We approach this task as a domain adaptation problem and
propose a self-training algorithm which imputes outcomes with categorical
values for finite unseen actions in the observational data to simulate a
randomized trial through pseudolabeling, which we refer to as Counterfactual
Self-Training (CST). CST iteratively imputes pseudolabels and retrains the
model. In addition, we show input consistency loss can further improve CST
performance which is shown in recent theoretical analysis of pseudolabeling. We
demonstrate the effectiveness of the proposed algorithms on both synthetic and
real datasets.
