  Biomedical summarization requires large datasets to train for text
generation. We show that while transfer learning offers a viable option for
addressing this challenge, an in-domain pre-training does not always offer
advantages in a BioASQ summarization task. We identify a suitable model
architecture and use it to show a benefit of a general-domain pre-training
followed by a task-specific fine-tuning in the context of a BioASQ
summarization task, leading to a novel three-step fine-tuning approach that
works with only a thousand in-domain examples. Our results indicate that a
Large Language Model without domain-specific pre-training can have a
significant edge in some domain-specific biomedical text generation tasks.
