  Human-robot interactive decision-making is increasingly becoming ubiquitous,
and trust is an influential factor in determining the reliance on autonomy.
However, it is not reasonable to trust systems that are beyond our
comprehension, and typical machine learning and data-driven decision-making are
black-box paradigms that impede interpretability. Therefore, it is critical to
establish computational trustworthy decision-making mechanisms enhanced by
interpretability-aware strategies. To this end, we propose a Trustworthy
Decision-Making (TDM) framework, which integrates symbolic planning into
sequential decision-making. The framework learns interpretable subtasks that
result in a complex, higher-level composite task that can be formally evaluated
using the proposed trust metric. TDM enables the subtask-level interpretability
by design and converges to an optimal symbolic plan from the learned subtasks.
Moreover, a TDM-based algorithm is introduced to demonstrate the unification of
symbolic planning with other sequential-decision making algorithms, reaping the
benefits of both. Experimental results validate the effectiveness of
trust-score-based planning while improving the interpretability of subtasks.
