  Autonomous vehicles (AVs) must interact with a diverse set of human drivers
in heterogeneous geographic areas. Ideally, fleets of AVs should share
trajectory data to continually re-train and improve trajectory forecasting
models from collective experience using cloud-based distributed learning. At
the same time, these robots should ideally avoid uploading raw driver
interaction data in order to protect proprietary policies (when sharing
insights with other companies) or protect driver privacy from insurance
companies. Federated learning (FL) is a popular mechanism to learn models in
cloud servers from diverse users without divulging private local data. However,
FL is often not robust -- it learns sub-optimal models when user data comes
from highly heterogeneous distributions, which is a key hallmark of human-robot
interactions. In this paper, we present a novel variant of personalized FL to
specialize robust robot learning models to diverse user distributions. Our
algorithm outperforms standard FL benchmarks by up to 2x in real user studies
that we conducted where human-operated vehicles must gracefully merge lanes
with simulated AVs in the standard CARLA and CARLO AV simulators.
