  Most real world language problems require learning from heterogenous corpora,
raising the problem of learning robust models which generalise well to both
similar (in domain) and dissimilar (out of domain) instances to those seen in
training. This requires learning an underlying task, while not learning
irrelevant signals and biases specific to individual domains. We propose a
novel method to optimise both in- and out-of-domain accuracy based on joint
learning of a structured neural model with domain-specific and domain-general
components, coupled with adversarial training for domain. Evaluating on
multi-domain language identification and multi-domain sentiment analysis, we
show substantial improvements over standard domain adaptation techniques, and
domain-adversarial training.
