  The segmentation of histopathological whole slide images into tumourous and
non-tumourous types of tissue is a challenging task that requires the
consideration of both local and global spatial contexts to classify tumourous
regions precisely. The identification of subtypes of tumour tissue complicates
the issue as the sharpness of separation decreases and the pathologist's
reasoning is even more guided by spatial context. However, the identification
of detailed types of tissue is crucial for providing personalized cancer
therapies. Due to the high resolution of whole slide images, existing semantic
segmentation methods, restricted to isolated image sections, are incapable of
processing context information beyond. To take a step towards better context
comprehension, we propose a patch neighbour attention mechanism to query the
neighbouring tissue context from a patch embedding memory bank and infuse
context embeddings into bottleneck hidden feature maps. Our memory attention
framework (MAF) mimics a pathologist's annotation procedure -- zooming out and
considering surrounding tissue context. The framework can be integrated into
any encoder-decoder segmentation method. We evaluate the MAF on a public breast
cancer and an internal kidney cancer data set using famous segmentation models
(U-Net, DeeplabV3) and demonstrate the superiority over other
context-integrating algorithms -- achieving a substantial improvement of up to
$17\%$ on Dice score. The code is publicly available at:
https://github.com/tio-ikim/valuing-vicinity
