  For medical imaging AI models to be clinically impactful, they must
generalize. However, this goal is hindered by (i) diverse types of distribution
shifts, such as temporal, demographic, and label shifts, and (ii) limited
diversity in datasets that are siloed within single medical institutions. While
these limitations have spurred interest in federated learning, current
evaluation benchmarks fail to evaluate different shifts simultaneously.
However, in real healthcare settings, multiple types of shifts co-exist, yet
their impact on medical imaging performance remains unstudied. In response, we
introduce FedMedICL, a unified framework and benchmark to holistically evaluate
federated medical imaging challenges, simultaneously capturing label,
demographic, and temporal distribution shifts. We comprehensively evaluate
several popular methods on six diverse medical imaging datasets (totaling 550
GPU hours). Furthermore, we use FedMedICL to simulate COVID-19 propagation
across hospitals and evaluate whether methods can adapt to pandemic changes in
disease prevalence. We find that a simple batch balancing technique surpasses
advanced methods in average performance across FedMedICL experiments. This
finding questions the applicability of results from previous, narrow benchmarks
in real-world medical settings.
