  We propose V2CNet, a new deep learning framework to automatically translate
the demonstration videos to commands that can be directly used in robotic
applications. Our V2CNet has two branches and aims at understanding the
demonstration video in a fine-grained manner. The first branch has the
encoder-decoder architecture to encode the visual features and sequentially
generate the output words as a command, while the second branch uses a Temporal
Convolutional Network (TCN) to learn the fine-grained actions. By jointly
training both branches, the network is able to model the sequential information
of the command, while effectively encodes the fine-grained actions. The
experimental results on our new large-scale dataset show that V2CNet
outperforms recent state-of-the-art methods by a substantial margin, while its
output can be applied in real robotic applications. The source code and trained
models will be made available.
