  Automated decision-making (ADM) systems are being deployed across a diverse
range of critical problem areas such as social welfare and healthcare. Recent
work highlights the importance of causal ML models in ADM systems, but
implementing them in complex social environments poses significant challenges.
Research on how these challenges impact the performance in specific downstream
decision-making tasks is limited. Addressing this gap, we make use of a
comprehensive real-world dataset of jobseekers to illustrate how the
performance of a single CATE model can vary significantly across different
decision-making scenarios and highlight the differential influence of
challenges such as distribution shifts on predictions and allocations.
