  The interaction of conversational systems with users poses an exciting
opportunity for improving them after deployment, but little evidence has been
provided of its feasibility. In most applications, users are not able to
provide the correct answer to the system, but they are able to provide binary
(correct, incorrect) feedback. In this paper we propose feedback-weighted
learning based on importance sampling to improve upon an initial supervised
system using binary user feedback. We perform simulated experiments on document
classification (for development) and Conversational Question Answering datasets
like QuAC and DoQA, where binary user feedback is derived from gold
annotations. The results show that our method is able to improve over the
initial supervised system, getting close to a fully-supervised system that has
access to the same labeled examples in in-domain experiments (QuAC), and even
matching in out-of-domain experiments (DoQA). Our work opens the prospect to
exploit interactions with real users and improve conversational systems after
deployment.
