Incorrigibility in the CIRL Framework
  A value learning system has incentives to follow shutdown instructions,
assuming the shutdown instruction provides information (in the technical sense)
about which actions lead to valuable outcomes. However, this assumption is not
robust to model mis-specification (e.g., in the case of programmer errors). We
demonstrate this by presenting some Supervised POMDP scenarios in which errors
in the parameterized reward function remove the incentive to follow shutdown
commands. These difficulties parallel those discussed by Soares et al. (2015)
in their paper on corrigibility. We argue that it is important to consider
systems that follow shutdown commands under some weaker set of assumptions
(e.g., that one small verified module is correctly implemented; as opposed to
an entire prior probability distribution and/or parameterized reward function).
We discuss some difficulties with simple ways to attempt to attain these sorts
of guarantees in a value learning framework.
