  Learning sentence vectors from an unlabeled corpus has attracted attention
because such vectors can represent sentences in a lower dimensional and
continuous space. Simple heuristics using pre-trained word vectors are widely
applied to machine learning tasks. However, they are not well understood from a
theoretical perspective. We analyze learning sentence vectors from a transfer
learning perspective by using a PAC-Bayes bound that enables us to understand
existing heuristics. We show that simple heuristics such as averaging and
inverse document frequency weighted averaging are derived by our formulation.
Moreover, we propose novel sentence vector learning algorithms on the basis of
our PAC-Bayes analysis.
