  Numerous advanced Large Language Models (LLMs) now support context lengths up
to 128K, and some extend to 200K. Some benchmarks in the generic domain have
also followed up on evaluating long-context capabilities. In the medical
domain, tasks are distinctive due to the unique contexts and need for domain
expertise, necessitating further evaluation. However, despite the frequent
presence of long texts in medical scenarios, evaluation benchmarks of
long-context capabilities for LLMs in this field are still rare. In this paper,
we propose MedOdyssey, the first medical long-context benchmark with seven
length levels ranging from 4K to 200K tokens. MedOdyssey consists of two
primary components: the medical-context "needles in a haystack" task and a
series of tasks specific to medical applications, together comprising 10
datasets. The first component includes challenges such as counter-intuitive
reasoning and novel (unknown) facts injection to mitigate knowledge leakage and
data contamination of LLMs. The second component confronts the challenge of
requiring professional medical expertise. Especially, we design the ``Maximum
Identical Context'' principle to improve fairness by guaranteeing that
different LLMs observe as many identical contexts as possible. Our experiment
evaluates advanced proprietary and open-source LLMs tailored for processing
long contexts and presents detailed performance analyses. This highlights that
LLMs still face challenges and need for further research in this area. Our code
and data are released in the repository:
\url{https://github.com/JOHNNY-fans/MedOdyssey.}
