  Smart cars, smartphones and other devices in the Internet of Things (IoT),
which usually have more than one sensors, produce multimodal data. Federated
Learning supports collecting a wealth of multimodal data from different devices
without sharing raw data. Transfer Learning methods help transfer knowledge
from some devices to others. Federated Transfer Learning methods benefit both
Federated Learning and Transfer Learning. This newly proposed Federated
Transfer Learning framework aims at connecting data islands with privacy
protection. Our construction is based on Federated Learning and Transfer
Learning. Compared with previous Federated Transfer Learnings, where each user
should have data with identical modalities (either all unimodal or all
multimodal), our new framework is more generic, it allows a hybrid distribution
of user data. The core strategy is to use two different but inherently
connected training methods for our two types of users. Supervised Learning is
adopted for users with only unimodal data (Type 1), while Self-Supervised
Learning is applied to user with multimodal data (Type 2) for both the feature
of each modality and the connection between them. This connection knowledge of
Type 2 will help Type 1 in later stages of training. Training in the new
framework can be divided in three steps. In the first step, users who have data
with the identical modalities are grouped together. For example, user with only
sound signals are in group one, and those with only images are in group two,
and users with multimodal data are in group three, and so on. In the second
step, Federated Learning is executed within the groups, where Supervised
Learning and Self-Supervised Learning are used depending on the group's nature.
Most of the Transfer Learning happens in the third step, where the related
parts in the network obtained from the previous steps are aggregated
(federated).
