  Monitoring patients in ICU is a challenging and high-cost task. Hence,
predicting the condition of patients during their ICU stay can help provide
better acute care and plan the hospital's resources. There has been continuous
progress in machine learning research for ICU management, and most of this work
has focused on using time series signals recorded by ICU instruments. In our
work, we show that adding clinical notes as another modality improves the
performance of the model for three benchmark tasks: in-hospital mortality
prediction, modeling decompensation, and length of stay forecasting that play
an important role in ICU management. While the time-series data is measured at
regular intervals, doctor notes are charted at irregular times, making it
challenging to model them together. We propose a method to model them jointly,
achieving considerable improvement across benchmark tasks over baseline
time-series model. Our implementation can be found at
\url{https://github.com/kaggarwal/ClinicalNotesICU}.
