  Human expectations arise from their understanding of others and the world. In
the context of human-AI interaction, this understanding may not align with
reality, leading to the AI agent failing to meet expectations and compromising
team performance. Explicable planning, introduced as a method to bridge this
gap, aims to reconcile human expectations with the agent's optimal behavior,
facilitating interpretable decision-making. However, an unresolved critical
issue is ensuring safety in explicable planning, as it could result in
explicable behaviors that are unsafe. To address this, we propose Safe
Explicable Planning (SEP), which extends the prior work to support the
specification of a safety bound. The goal of SEP is to find behaviors that
align with human expectations while adhering to the specified safety criterion.
Our approach generalizes the consideration of multiple objectives stemming from
multiple models rather than a single model, yielding a Pareto set of safe
explicable policies. We present both an exact method, guaranteeing finding the
Pareto set, and a more efficient greedy method that finds one of the policies
in the Pareto set. Additionally, we offer approximate solutions based on state
aggregation to improve scalability. We provide formal proofs that validate the
desired theoretical properties of these methods. Evaluation through simulations
and physical robot experiments confirms the effectiveness of our approach for
safe explicable planning.
