  As assistive and collaborative robots become more ubiquitous in the
real-world, we need to develop interfaces and controllers that are safe for
users to build trust and encourage adoption. In this Blue Sky paper, we discuss
the need for co-evolving task and user-specific safety controllers that can
accommodate people's safety preferences. We argue that while most adaptive
controllers focus on behavioral adaptation, safety adaptation is also a major
consideration for building trust in collaborative systems. Furthermore, we
highlight the need for adaptation over time, to account for user's changes in
preferences as experience and trust builds. We provide a general formulation
for what these interfaces should look like and what features are necessary for
making them feasible and successful. In this formulation, users provide
demonstrations and labelled safety ratings from which a safety value function
is learned. These value functions can be updated by updating the safety labels
on demonstrations to learn an updated function. We discuss how this can be
implemented at a high-level, as well as some promising approaches and
techniques for enabling this.
