  Electronic Health Records (EHR) data contains medical records such as
diagnoses, medications, procedures, and treatments of patients. This data is
often considered sensitive medical information. Therefore, the EHR data from
the medical centers often cannot be shared, making it difficult to create
prediction models using multi-center EHR data, which is essential for such
models' robustness and generalizability. Federated Learning (FL) is an
algorithmic approach that allows learning a shared model using data in multiple
locations without the need to store all data in a central place. An example of
a prediction model's task is to predict future diseases. More specifically, the
model needs to predict patient's next visit diagnoses, based on current and
previous clinical data. Such a prediction model can support care providers in
making clinical decisions and even provide preventive treatment. We propose a
federated learning approach for learning medical concepts embedding. This
pre-trained model can be used for fine-tuning for specific downstream tasks.
Our approach is based on an embedding model like BEHRT, a deep neural sequence
transduction model for EHR. We train using federated learning, both the Masked
Language Modeling (MLM) and the next visit downstream model. We demonstrate our
approach on the MIMIC-IV dataset. We compare the performance of a model trained
with FL against a model trained on centralized data. We find that our federated
learning approach reaches very close to the performance of a centralized model,
and it outperforms local models in terms of average precision. We also show
that pre-trained MLM improves the model's average precision performance in the
next visit prediction task, compared to an MLM model without pre-training. Our
code is available at https://github.com/nadavlab/FederatedBEHRT.
