  Effective representation learning of electronic health records is a
challenging task and is becoming more important as the availability of such
data is becoming pervasive. The data contained in these records are irregular
and contain multiple modalities such as notes, and medical codes. They are
preempted by medical conditions the patient may have, and are typically jotted
down by medical staff. Accompanying codes are notes containing valuable
information about patients beyond the structured information contained in
electronic health records. We use transformer networks and the recently
proposed BERT language model to embed these data streams into a unified vector
representation. The presented approach effectively encodes a patient's visit
data into a single distributed representation, which can be used for downstream
tasks. Our model demonstrates superior performance and generalization on
mortality, readmission and length of stay tasks using the publicly available
MIMIC-III ICU dataset. Code avaialble at
https://github.com/sajaddarabi/TAPER-EHR
