  Deep learning for histopathology has been successfully used for disease
classification, image segmentation and more. However, combining image and text
modalities using current state-of-the-art (SOTA) methods has been a challenge
due to the high resolution of histopathology images. Automatic report
generation for histopathology images is one such challenge. In this work, we
show that using an existing pre-trained Vision Transformer (ViT) to encode
4096x4096 sized patches of the Whole Slide Image (WSI) and a pre-trained
Bidirectional Encoder Representations from Transformers (BERT) model for
language modeling-based decoder for report generation, we can build a
performant and portable report generation mechanism that takes into account the
whole high resolution image. Our method allows us to not only generate and
evaluate captions that describe the image, but also helps us classify the image
into tissue types and the gender of the patient as well. Our best performing
model achieves a 89.52% accuracy in Tissue Type classification with a BLEU-4
score of 0.12 in our caption generation task.
