  This study achieved bidirectional translation between descriptions and
actions using small paired data from different modalities. The ability to
mutually generate descriptions and actions is essential for robots to
collaborate with humans in their daily lives, which generally requires a large
dataset that maintains comprehensive pairs of both modality data. However, a
paired dataset is expensive to construct and difficult to collect. To address
this issue, this study proposes a two-stage training method for bidirectional
translation. In the proposed method, we train recurrent autoencoders (RAEs) for
descriptions and actions with a large amount of non-paired data. Then, we
finetune the entire model to bind their intermediate representations using
small paired data. Because the data used for pre-training do not require
pairing, behavior-only data or a large language corpus can be used. We
experimentally evaluated our method using a paired dataset consisting of
motion-captured actions and descriptions. The results showed that our method
performed well, even when the amount of paired data to train was small. The
visualization of the intermediate representations of each RAE showed that
similar actions were encoded in a clustered position and the corresponding
feature vectors were well aligned.
