  In this paper, we address the task of learning novel visual concepts, and
their interactions with other concepts, from a few images with sentence
descriptions. Using linguistic context and visual features, our method is able
to efficiently hypothesize the semantic meaning of new words and add them to
its word dictionary so that they can be used to describe images which contain
these novel concepts. Our method has an image captioning module based on m-RNN
with several improvements. In particular, we propose a transposed weight
sharing scheme, which not only improves performance on image captioning, but
also makes the model more suitable for the novel concept learning task. We
propose methods to prevent overfitting the new concepts. In addition, three
novel concept datasets are constructed for this new task. In the experiments,
we show that our method effectively learns novel visual concepts from a few
examples without disturbing the previously learned concepts. The project page
is http://www.stat.ucla.edu/~junhua.mao/projects/child_learning.html
