  In medical applications, deep learning methods are built to automate
diagnostic tasks. However, a clinically relevant question that practitioners
usually face, is how to predict the future trajectory of a disease (prognosis).
Current methods for such a problem often require domain knowledge, and are
complicated to apply. In this paper, we formulate the prognosis prediction
problem as a one-to-many forecasting problem from multimodal data. Inspired by
a clinical decision-making process with two agents -- a radiologist and a
general practitioner, we model a prognosis prediction problem with two
transformer-based components that share information between each other. The
first block in this model aims to analyze the imaging data, and the second
block leverages the internal representations of the first one as inputs, also
fusing them with auxiliary patient data. We show the effectiveness of our
method in predicting the development of structural knee osteoarthritis changes
over time. Our results show that the proposed method outperforms the
state-of-the-art baselines in terms of various performance metrics. In
addition, we empirically show that the existence of the multi-agent
transformers with depths of 2 is sufficient to achieve good performances. Our
code is publicly available at \url{https://github.com/MIPT-Oulu/CLIMAT}.
