  Hematoxylin and Eosin stained histopathology image analysis is essential for
the diagnosis and study of complicated diseases such as cancer. Existing
state-of-the-art approaches demand extensive amount of supervised training data
from trained pathologists. In this work we synthesize in an unsupervised
manner, large histopathology image datasets, suitable for supervised training
tasks. We propose a unified pipeline that: a) generates a set of initial
synthetic histopathology images with paired information about the nuclei such
as segmentation masks; b) refines the initial synthetic images through a
Generative Adversarial Network (GAN) to reference styles; c) trains a
task-specific CNN and boosts the performance of the task-specific CNN with
on-the-fly generated adversarial examples. Our main contribution is that the
synthetic images are not only realistic, but also representative (in reference
styles) and relatively challenging for training task-specific CNNs. We test our
method for nucleus segmentation using images from four cancer types. When no
supervised data exists for a cancer type, our method without supervision cost
significantly outperforms supervised methods which perform across-cancer
generalization. Even when supervised data exists for all cancer types, our
approach without supervision cost performs better than supervised methods.
