  Neural Architecture Search (NAS) methods, which automatically learn entire
neural model or individual neural cell architectures, have recently achieved
competitive or state-of-the-art (SOTA) performance on variety of natural
language processing and computer vision tasks, including language modeling,
natural language inference, and image classification. In this work, we explore
the applicability of a SOTA NAS algorithm, Efficient Neural Architecture Search
(ENAS) (Pham et al., 2018) to two sentence pair tasks, paraphrase detection and
semantic textual similarity. We use ENAS to perform a micro-level search and
learn a task-optimized RNN cell architecture as a drop-in replacement for an
LSTM. We explore the effectiveness of ENAS through experiments on three
datasets (MRPC, SICK, STS-B), with two different models (ESIM, BiLSTM-Max), and
two sets of embeddings (Glove, BERT). In contrast to prior work applying ENAS
to NLP tasks, our results are mixed -- we find that ENAS architectures
sometimes, but not always, outperform LSTMs and perform similarly to random
architecture search.
