  In recognition-based action interaction, robots' responses to human actions
are often pre-designed according to recognized categories and thus stiff. In
this paper, we specify a new Interactive Action Translation (IAT) task which
aims to learn end-to-end action interaction from unlabeled interactive pairs,
removing explicit action recognition. To enable learning on small-scale data,
we propose a Paired-Embedding (PE) method for effective and reliable data
augmentation. Specifically, our method first utilizes paired relationships to
cluster individual actions in an embedding space. Then two actions originally
paired can be replaced with other actions in their respective neighborhood,
assembling into new pairs. An Act2Act network based on conditional GAN follows
to learn from augmented data. Besides, IAT-test and IAT-train scores are
specifically proposed for evaluating methods on our task. Experimental results
on two datasets show impressive effects and broad application prospects of our
method.
