  In transfer learning, it is imperative to achieve strong alignment between a
pre-trained model and a downstream task. Prior work has done this by proposing
task-specific pre-training objectives, which sacrifices the inherent
scalability of the transfer learning paradigm. We instead achieve strong
alignment by simultaneously modifying both the pre-trained model and the
formulation of the downstream task, which is more efficient and preserves the
scalability of transfer learning. We present GenSF (Generative Slot Filling),
which leverages a generative pre-trained open-domain dialog model for slot
filling. GenSF (1) adapts the pre-trained model by incorporating inductive
biases about the task and (2) adapts the downstream task by reformulating slot
filling to better leverage the pre-trained model's capabilities. GenSF achieves
state-of-the-art results on two slot filling datasets with strong gains in
few-shot and zero-shot settings. We achieve a 9 F1 score improvement in
zero-shot slot filling. This highlights the value of strong alignment between
the pre-trained model and the downstream task.
