  Safe control methods are often intended to behave safely even in worst-case
human uncertainties. However, humans may exploit such safety-first systems,
which results in greater risk for everyone. Despite their significance, no
prior work has investigated and accounted for such factors in safe control. In
this paper, we leverage an interaction-based payoff structure from game theory
to model humans' short-sighted, self-seeking behaviors and how humans change
their strategies toward machines based on prior experience. We integrate such
strategic human behaviors into a safe control architecture. As a result, our
approach achieves better safety and performance trade-offs when compared to
both deterministic worst-case safe control techniques and equilibrium-based
stochastic methods. Our findings suggest an urgent need to fundamentally
rethink the safe control framework used in human-technology interaction in
pursuit of greater safety for all.
