  We introduce ViFiCon, a self-supervised contrastive learning scheme which
uses synchronized information across vision and wireless modalities to perform
cross-modal association. Specifically, the system uses pedestrian data
collected from RGB-D camera footage as well as WiFi Fine Time Measurements
(FTM) from a user's smartphone device. We represent the temporal sequence by
stacking multi-person depth data spatially within a banded image. Depth data
from RGB-D (vision domain) is inherently linked with an observable pedestrian,
but FTM data (wireless domain) is associated only to a smartphone on the
network. To formulate the cross-modal association problem as self-supervised,
the network learns a scene-wide synchronization of the two modalities as a
pretext task, and then uses that learned representation for the downstream task
of associating individual bounding boxes to specific smartphones, i.e.
associating vision and wireless information. We use a pre-trained region
proposal model on the camera footage and then feed the extrapolated bounding
box information into a dual-branch convolutional neural network along with the
FTM data. We show that compared to fully supervised SoTA models, ViFiCon
achieves high performance vision-to-wireless association, finding which
bounding box corresponds to which smartphone device, without hand-labeled
association examples for training data.
