  In many risk-aware and multi-objective reinforcement learning settings, the
utility of the user is derived from a single execution of a policy. In these
settings, making decisions based on the average future returns is not suitable.
For example, in a medical setting a patient may only have one opportunity to
treat their illness. Making decisions using just the expected future returns --
known in reinforcement learning as the value -- cannot account for the
potential range of adverse or positive outcomes a decision may have. Therefore,
we should use the distribution over expected future returns differently to
represent the critical information that the agent requires at decision time by
taking both the future and accrued returns into consideration. In this paper,
we propose two novel Monte Carlo tree search algorithms. Firstly, we present a
Monte Carlo tree search algorithm that can compute policies for nonlinear
utility functions (NLU-MCTS) by optimising the utility of the different
possible returns attainable from individual policy executions, resulting in
good policies for both risk-aware and multi-objective settings. Secondly, we
propose a distributional Monte Carlo tree search algorithm (DMCTS) which
extends NLU-MCTS. DMCTS computes an approximate posterior distribution over the
utility of the returns, and utilises Thompson sampling during planning to
compute policies in risk-aware and multi-objective settings. Both algorithms
outperform the state-of-the-art in multi-objective reinforcement learning for
the expected utility of the returns.
