  We propose ConVEx (Conversational Value Extractor), an efficient pretraining
and fine-tuning neural approach for slot-labeling dialog tasks. Instead of
relying on more general pretraining objectives from prior work (e.g., language
modeling, response selection), ConVEx's pretraining objective, a novel pairwise
cloze task using Reddit data, is well aligned with its intended usage on
sequence labeling tasks. This enables learning domain-specific slot labelers by
simply fine-tuning decoding layers of the pretrained general-purpose sequence
labeling model, while the majority of the pretrained model's parameters are
kept frozen. We report state-of-the-art performance of ConVEx across a range of
diverse domains and data sets for dialog slot-labeling, with the largest gains
in the most challenging, few-shot setups. We believe that ConVEx's reduced
pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its
efficient fine-tuning and strong performance, promise wider portability and
scalability for data-efficient sequence-labeling tasks in general.
