  Recent advances with language models (e.g. BERT, XLNet, ...), have allowed
surpassing human performance on complex NLP tasks such as Reading
Comprehension. However, labeled datasets for training are available mostly in
English which makes it difficult to acknowledge progress in other languages.
Fortunately, models are now pre-trained on unlabeled data from hundreds of
languages and exhibit interesting transfer abilities from one language to
another. In this paper, we show that multilingual BERT is naturally capable of
zero-shot transfer for an extractive Question Answering task (eQA) from English
to other languages. More specifically, it outperforms the best previously known
baseline for transfer to Japanese and French. Moreover, using a recently
published large eQA French dataset, we are able to further show that (1)
zero-shot transfer provides results really close to a direct training on the
target language and (2) combination of transfer and training on target is the
best option overall. We finally present a practical application: a multilingual
conversational agent called Kate which answers to HR-related questions in
several languages directly from the content of intranet pages.
