  Neural network pretraining is gaining attention due to its outstanding
performance in natural language processing applications. However, pretraining
usually leverages predefined task sequences to learn general linguistic clues.
The lack of mechanisms in choosing proper tasks during pretraining makes the
learning and knowledge encoding inefficient. We thus propose using
meta-learning to select tasks that provide the most informative learning
signals in each episode of pretraining. With the proposed method, we aim to
achieve better efficiency in computation and memory usage for the pretraining
process and resulting networks while maintaining the performance. In this
preliminary work, we discuss the algorithm of the method and its two variants,
downstream-aware and downstream-agnostic pretraining. Our experiment plan is
also summarized, while empirical results will be shared in our future works.
