  Motivated by the human-machine interaction such as training chatbots for
improving customer satisfaction, we study human-guided human-machine
interaction involving private information. We model this interaction as a
two-player turn-based game, where one player (Alice, a human) guides the other
player (Bob, a machine) towards a common goal. Specifically, we focus on
offline reinforcement learning (RL) in this game, where the goal is to find a
policy pair for Alice and Bob that maximizes their expected total rewards based
on an offline dataset collected a priori. The offline setting presents two
challenges: (i) We cannot collect Bob's private information, leading to a
confounding bias when using standard RL methods, and (ii) a distributional
mismatch between the behavior policy used to collect data and the desired
policy we aim to learn. To tackle the confounding bias, we treat Bob's previous
action as an instrumental variable for Alice's current decision making so as to
adjust for the unmeasured confounding. We develop a novel identification result
and use it to propose a new off-policy evaluation (OPE) method for evaluating
policy pairs in this two-player turn-based game. To tackle the distributional
mismatch, we leverage the idea of pessimism and use our OPE method to develop
an off-policy learning algorithm for finding a desirable policy pair for both
Alice and Bob. Finally, we prove that under mild assumptions such as partial
coverage of the offline data, the policy pair obtained through our method
converges to the optimal one at a satisfactory rate.
