  Domain adaptation allows generative language models to address specific flaws
caused by the domain shift of their application. However, the traditional
adaptation by further training on in-domain data rapidly weakens the model's
ability to generalize to other domains, making the open-ended deployments of
the adapted models prone to errors. This work introduces novel training
objectives built upon a semantic similarity of the predicted tokens to the
reference.
  Our results show that (1) avoiding the common assumption of a single correct
prediction by constructing the training target from tokens' semantic similarity
can mitigate catastrophic forgetting during domain adaptation, while (2)
preserving the quality of the adaptation, (3) with negligible additions to
compute costs.
  In the broader context, the objectives grounded in a continuous token
similarity pioneer the exploration of the middle ground between the efficient
but na\"{\i}ve exact-match token-level objectives and expressive but
computationally- and resource-intensive sequential objectives.
