  In this paper we study how different ways of combining character and
word-level representations affect the quality of both final word and sentence
representations. We provide strong empirical evidence that modeling characters
improves the learned representations at the word and sentence levels, and that
doing so is particularly useful when representing less frequent words. We
further show that a feature-wise sigmoid gating mechanism is a robust method
for creating representations that encode semantic similarity, as it performed
reasonably well in several word similarity datasets. Finally, our findings
suggest that properly capturing semantic similarity at the word level does not
consistently yield improved performance in downstream sentence-level tasks. Our
code is available at https://github.com/jabalazs/gating
