  Traditional information retrieval (such as that offered by web search
engines) impedes users with information overload from extensive result pages
and the need to manually locate the desired information therein. Conversely,
question-answering systems change how humans interact with information systems:
users can now ask specific questions and obtain a tailored answer - both
conveniently in natural language. Despite obvious benefits, their use is often
limited to an academic context, largely because of expensive domain
customizations, which means that the performance in domain-specific
applications often fails to meet expectations. This paper proposes
cost-efficient remedies: (i) we leverage metadata through a filtering
mechanism, which increases the precision of document retrieval, and (ii) we
develop a novel fuse-and-oversample approach for transfer learning in order to
improve the performance of answer extraction. Here knowledge is inductively
transferred from a related, yet different, tasks to the domain-specific
application, while accounting for potential differences in the sample sizes
across both tasks. The resulting performance is demonstrated with actual use
cases from a finance company and the film industry, where fewer than 400
question-answer pairs had to be annotated in order to yield significant
performance gains. As a direct implication to management, this presents a
promising path to better leveraging of knowledge stored in information systems.
