  Deep learning methods such as convolutional neural networks (CNNs) are
difficult to directly utilize to analyze whole slide images (WSIs) due to the
large image dimensions. We overcome this limitation by proposing a novel
two-stage approach. First, we extract a set of representative patches (called
mosaic) from a WSI. Each patch of a mosaic is encoded to a feature vector using
a deep network. The feature extractor model is fine-tuned using hierarchical
target labels of WSIs, i.e., anatomic site and primary diagnosis. In the second
stage, a set of encoded patch-level features from a WSI is used to compute the
primary diagnosis probability through the proposed Pay Attention with Focus
scheme, an attention-weighted averaging of predicted probabilities for all
patches of a mosaic modulated by a trainable focal factor. Experimental results
show that the proposed model can be robust, and effective for the
classification of WSIs.
