  A major hurdle on the road to conversational interfaces is the difficulty in
collecting data that maps language utterances to logical forms. One prominent
approach for data collection has been to automatically generate pseudo-language
paired with logical forms, and paraphrase the pseudo-language to natural
language through crowdsourcing (Wang et al., 2015). However, this data
collection procedure often leads to low performance on real data, due to a
mismatch between the true distribution of examples and the distribution induced
by the data collection procedure. In this paper, we thoroughly analyze two
sources of mismatch in this process: the mismatch in logical form distribution
and the mismatch in language distribution between the true and induced
distributions. We quantify the effects of these mismatches, and propose a new
data collection approach that mitigates them. Assuming access to unlabeled
utterances from the true distribution, we combine crowdsourcing with a
paraphrase model to detect correct logical forms for the unlabeled utterances.
On two datasets, our method leads to 70.6 accuracy on average on the true
distribution, compared to 51.3 in paraphrasing-based data collection.
