  Domain shifts, such as appearance changes, are a key challenge in real-world
applications of activity recognition models, which range from assistive
robotics and smart homes to driver observation in intelligent vehicles. For
example, while simulations are an excellent way of economical data collection,
a Synthetic-to-Real domain shift leads to a > 60% drop in accuracy when
recognizing activities of Daily Living (ADLs). We tackle this challenge and
introduce an activity domain generation framework which creates novel ADL
appearances (novel domains) from different existing activity modalities (source
domains) inferred from video training data. Our framework computes human poses,
heatmaps of body joints, and optical flow maps and uses them alongside the
original RGB videos to learn the essence of source domains in order to generate
completely new ADL domains. The model is optimized by maximizing the distance
between the existing source appearances and the generated novel appearances
while ensuring that the semantics of an activity is preserved through an
additional classification loss. While source data multimodality is an important
concept in this design, our setup does not rely on multi-sensor setups, (i.e.,
all source modalities are inferred from a single video only.) The newly created
activity domains are then integrated in the training of the ADL classification
networks, resulting in models far less susceptible to changes in data
distributions. Extensive experiments on the Synthetic-to-Real benchmark
Sims4Action demonstrate the potential of the domain generation paradigm for
cross-domain ADL recognition, setting new state-of-the-art results. Our code is
publicly available at https://github.com/Zrrr1997/syn2real_DG
