  Given a user's input text, text-matching recommender systems output relevant
items by comparing the input text to available items' description, such as
product-to-product recommendation on e-commerce platforms. As users' interests
and item inventory are expected to change, it is important for a text-matching
system to generalize to data shifts, a task known as out-of-distribution (OOD)
generalization. However, we find that the popular approach of fine-tuning a
large, base language model on paired item relevance data (e.g., user clicks)
can be counter-productive for OOD generalization. For a product recommendation
task, fine-tuning obtains worse accuracy than the base model when recommending
items in a new category or for a future time period. To explain this
generalization failure, we consider an intervention-based importance metric,
which shows that a fine-tuned model captures spurious correlations and fails to
learn the causal features that determine the relevance between any two text
inputs. Moreover, standard methods for causal regularization do not apply in
this setting, because unlike in images, there exist no universally spurious
features in a text-matching task (the same token may be spurious or causal
depending on the text it is being matched to). For OOD generalization on text
inputs, therefore, we highlight a different goal: avoiding high importance
scores for certain features. We do so using an intervention-based regularizer
that constraints the causal effect of any token on the model's relevance score
to be similar to the base model. Results on Amazon product and 3 question
recommendation datasets show that our proposed regularizer improves
generalization for both in-distribution and OOD evaluation, especially in
difficult scenarios when the base model is not accurate.
