  People can acquire knowledge in an unsupervised manner by reading, and
compose the knowledge to make novel combinations. In this paper, we investigate
whether pretrained language models can perform compositional generalization in
a realistic setting: recipe generation. We design the counterfactual recipe
generation task, which asks models to modify a base recipe according to the
change of an ingredient. This task requires compositional generalization at two
levels: the surface level of incorporating the new ingredient into the base
recipe, and the deeper level of adjusting actions related to the changing
ingredient. We collect a large-scale recipe dataset in Chinese for models to
learn culinary knowledge, and a subset of action-level fine-grained annotations
for evaluation. We finetune pretrained language models on the recipe corpus,
and use unsupervised counterfactual generation methods to generate modified
recipes. Results show that existing models have difficulties in modifying the
ingredients while preserving the original text style, and often miss actions
that need to be adjusted. Although pretrained language models can generate
fluent recipe texts, they fail to truly learn and use the culinary knowledge in
a compositional way. Code and data are available at
https://github.com/xxxiaol/counterfactual-recipe-generation.
