  The pretraining-fine-tuning paradigm has been the de facto strategy for
transfer learning in modern language modeling. With the understanding that task
adaptation in LMs is often a function of parameters shared across tasks, we
argue that a more surgical approach to regularization needs to exist for
smoother transfer learning. Towards this end, we investigate how the
pretraining loss landscape is affected by these task-sensitive parameters
through an information-theoretic lens. We then leverage the findings from our
investigations to devise a novel approach to dropout for improved model
regularization and better downstream generalization. This approach, named
guided dropout, is both task & architecture agnostic and adds no computational
overhead to the fine-tuning process. Through empirical evaluations, we showcase
that our approach to regularization yields consistently better performance,
even in scenarios of data paucity, compared to standardized baselines.
