  One of the goals of natural language understanding is to develop models that
map sentences into meaning representations. However, training such models
requires expensive annotation of complex structures, which hinders their
adoption. Learning to actively-learn (LTAL) is a recent paradigm for reducing
the amount of labeled data by learning a policy that selects which samples
should be labeled. In this work, we examine LTAL for learning semantic
representations, such as QA-SRL. We show that even an oracle policy that is
allowed to pick examples that maximize performance on the test set (and
constitutes an upper bound on the potential of LTAL), does not substantially
improve performance compared to a random policy. We investigate factors that
could explain this finding and show that a distinguishing characteristic of
successful applications of LTAL is the interaction between optimization and the
oracle policy selection process. In successful applications of LTAL, the
examples selected by the oracle policy do not substantially depend on the
optimization procedure, while in our setup the stochastic nature of
optimization strongly affects the examples selected by the oracle. We conclude
that the current applicability of LTAL for improving data efficiency in
learning semantic meaning representations is limited.
