  Deep learning has become the workhorse for a wide range of natural language
processing applications. But much of the success of deep learning relies on
annotated examples. Annotation is time-consuming and expensive to produce at
scale. Here we are interested in methods for reducing the required quantity of
annotated data -- by making the learning methods more knowledge efficient so as
to make them more applicable in low annotation (low resource) settings. There
are various classical approaches to making the models more knowledge efficient
such as multi-task learning, transfer learning, weakly supervised and
unsupervised learning etc. This thesis focuses on adapting such classical
methods to modern deep learning models and algorithms.
  This thesis describes four works aimed at making machine learning models more
knowledge efficient. First, we propose a knowledge rich deep learning model
(KRDL) as a unifying learning framework for incorporating prior knowledge into
deep models. In particular, we apply KRDL built on Markov logic networks to
denoise weak supervision. Second, we apply a KRDL model to assist the machine
reading models to find the correct evidence sentences that can support their
decision. Third, we investigate the knowledge transfer techniques in
multilingual setting, where we proposed a method that can improve pre-trained
multilingual BERT based on the bilingual dictionary. Fourth, we present an
episodic memory network for language modelling, in which we encode the large
external knowledge for the pre-trained GPT.
