  In this work, we modify and apply self-supervision techniques to the domain
of medical health insurance claims. We model patients' healthcare claims
history analogous to free-text narratives, and introduce pre-trained `prior
knowledge', later utilized for patient outcome predictions on a challenging
task: predicting Covid-19 hospitalization, given a patient's pre-Covid-19
insurance claims history. Results suggest that pre-training on insurance claims
not only produces better prediction performance, but, more importantly,
improves the model's `clinical trustworthiness' and model
stability/reliability.
