  Large Language Models (LLMs) have demonstrated remarkable capabilities across
various domains and are moving towards more specialized areas. Recent advanced
proprietary models such as GPT-4 and Gemini have achieved significant
advancements in biomedicine, which have also raised privacy and security
challenges. The construction of specialized generalists hinges largely on
high-quality datasets, enhanced by techniques like supervised fine-tuning and
reinforcement learning from human or AI feedback, and direct preference
optimization. However, these leading technologies (e.g., preference learning)
are still significantly limited in the open source community due to the
scarcity of specialized data. In this paper, we present the UltraMedical
collections, which consist of high-quality manual and synthetic datasets in the
biomedicine domain, featuring preference annotations across multiple advanced
LLMs. By utilizing these datasets, we fine-tune a suite of specialized medical
models based on Llama-3 series, demonstrating breathtaking capabilities across
various medical benchmarks. Moreover, we develop powerful reward models skilled
in biomedical and general reward benchmark, enhancing further online preference
learning within the biomedical LLM community.
