  Artificial Intelligence holds tremendous potential in medicine, but is
traditionally limited by the lack of massive datasets to train models on.
Foundation models, pre-trained models that can be adapted to downstream tasks
with small datasets, could alleviate this problem. Researchers at Moorfields
Eye Hospital (MEH) proposed RETFound-MEH, a foundation model for retinal
imaging that was trained on 900,000 images, including private hospital data.
Recently, data-efficient DERETFound was proposed that provides comparable
performance while being trained on only 150,000 images that are all publicly
available. However, both these models required very substantial resources to
train initially and are resource-intensive in downstream use. We propose a
novel Token Reconstruction objective that we use to train RETFound-Green, a
retinal foundation model trained using only 75,000 publicly available images
and 400 times less compute. We estimate the cost of training RETFound-MEH and
DERETFound at $10,000 and $14,000, respectively, while RETFound-Green could be
trained for less than $100, with equally reduced environmental impact.
RETFound-Green is also far more efficient in downstream use: it can be
downloaded 14 times faster, computes vector embeddings 2.7 times faster which
then require 2.6 times less storage space. Despite this, RETFound-Green does
not perform systematically worse. In fact, it performs best on 14 tasks,
compared to six for DERETFound and two for RETFound-MEH. Our results suggest
that RETFound-Green is a very efficient, high-performance retinal foundation
model. We anticipate that our Token Reconstruction objective could be scaled up
for even higher performance and be applied to other domains beyond retinal
imaging.
