  Clinical diagnosis of the eye is performed over multifarious data modalities
including scalar clinical labels, vectorized biomarkers, two-dimensional fundus
images, and three-dimensional Optical Coherence Tomography (OCT) scans.
Clinical practitioners use all available data modalities for diagnosing and
treating eye diseases like Diabetic Retinopathy (DR) or Diabetic Macular Edema
(DME). Enabling usage of machine learning algorithms within the ophthalmic
medical domain requires research into the relationships and interactions
between all relevant data over a treatment period. Existing datasets are
limited in that they neither provide data nor consider the explicit
relationship modeling between the data modalities. In this paper, we introduce
the Ophthalmic Labels for Investigating Visual Eye Semantics (OLIVES) dataset
that addresses the above limitation. This is the first OCT and near-IR fundus
dataset that includes clinical labels, biomarker labels, disease labels, and
time-series patient treatment information from associated clinical trials. The
dataset consists of 1268 near-IR fundus images each with at least 49 OCT scans,
and 16 biomarkers, along with 4 clinical labels and a disease diagnosis of DR
or DME. In total, there are 96 eyes' data averaged over a period of at least
two years with each eye treated for an average of 66 weeks and 7 injections. We
benchmark the utility of OLIVES dataset for ophthalmic data as well as provide
benchmarks and concrete research directions for core and emerging machine
learning paradigms within medical image analysis.
