  Counterfactual explanations (CFEs) are a popular approach in explainable
artificial intelligence (xAI), highlighting changes to input data necessary for
altering a model's output. A CFE can either describe a scenario that is better
than the factual state (upward CFE), or a scenario that is worse than the
factual state (downward CFE). However, potential benefits and drawbacks of the
directionality of CFEs for user behavior in xAI remain unclear. The current
user study (N=161) compares the impact of CFE directionality on behavior and
experience of participants tasked to extract new knowledge from an automated
system based on model predictions and CFEs. Results suggest that upward CFEs
provide a significant performance advantage over other forms of counterfactual
feedback. Moreover, the study highlights potential benefits of mixed CFEs
improving user performance compared to downward CFEs or no explanations. In
line with the performance results, users' explicit knowledge of the system is
statistically higher after receiving upward CFEs compared to downward
comparisons. These findings imply that the alignment between explanation and
task at hand, the so-called regulatory fit, may play a crucial role in
determining the effectiveness of model explanations, informing future research
directions in xAI. To ensure reproducible research, the entire code, underlying
models and user data of this study is openly available:
https://github.com/ukuhl/DirectionalAlienZoo
