  Training end-to-end networks for classifying gigapixel size histopathological
images is computationally intractable. Most approaches are patch-based and
first learn local representations (patch-wise) before combining these local
representations to produce image-level decisions. However, dividing large
tissue structures into patches limits the context available to these networks,
which may reduce their ability to learn representations from clinically
relevant structures. In this paper, we introduce a novel attention-based
network, the Holistic ATtention Network (HATNet) to classify breast biopsy
images. We streamline the histopathological image classification pipeline and
show how to learn representations from gigapixel size images end-to-end. HATNet
extends the bag-of-words approach and uses self-attention to encode global
information, allowing it to learn representations from clinically relevant
tissue structures without any explicit supervision. It outperforms the previous
best network Y-Net, which uses supervision in the form of tissue-level
segmentation masks, by 8%. Importantly, our analysis reveals that HATNet learns
representations from clinically relevant structures, and it matches the
classification accuracy of human pathologists for this challenging test set.
Our source code is available at \url{https://github.com/sacmehta/HATNet}
