  Learning-based approaches, such as reinforcement and imitation learning are
gaining popularity in decision-making for autonomous driving. However, learned
policies often fail to generalize and cannot handle novel situations well.
Asking and answering questions in the form of "Would a policy perform well if
the other agents had behaved differently?" can shed light on whether a policy
has seen similar situations during training and generalizes well. In this work,
a counterfactual policy evaluation is introduced that makes use of
counterfactual worlds - worlds in which the behaviors of others are non-actual.
If a policy can handle all counterfactual worlds well, it either has seen
similar situations during training or it generalizes well and is deemed to be
fit enough to be executed in the actual world. Additionally, by performing the
counterfactual policy evaluation, causal relations and the influence of
changing vehicle's behaviors on the surrounding vehicles becomes evident. To
validate the proposed method, we learn a policy using reinforcement learning
for a lane merging scenario. In the application-phase, the policy is only
executed after the counterfactual policy evaluation has been performed and if
the policy is found to be safe enough. We show that the proposed approach
significantly decreases the collision-rate whilst maintaining a high
success-rate.
