Skin Cancer Detection and Tracking\\using Data Synthesis and Deep Learning
Yunzhu Li\thanks{indicate equal contribution.
% Dense object detection and temporal tracking is needed across applications domains ranging from people-tracking to analysis of satellite imagery over time. The detection and tracking of malignant skin cancers and benign moles poses a particularly challenging problem due to the general uniformity of large skin patches, the fact that skin lesions vary little in their appearance, and the relatively small amount of data available. Here we introduce a novel data synthesis technique that merges images of individual skin lesions with full-body images and heavily augments them to generate significant amounts of data. We build a convolutional neural network (CNN) based system, trained on this synthetic data, and demonstrate superior performance to traditional detection and tracking techniques. Our system is intended for potential clinical use to augment the capabilities of healthcare providers. While domain-specific, we believe the methods invoked in this work will be useful in applying CNNs across domains that suffer from limited data availability. %
main.tex
\documentclass{article}

% if you need to pass options to natbib, use, e.g.:
\PassOptionsToPackage{numbers, compress}{natbib}
% before loading nips_2016
%
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{nips_2016}

%\usepackage{nips_2016}

% to compile a camera-ready version, add the [final] option, e.g.:
\usepackage[final]{nips_2016}
\usepackage{natbib}
\bibliographystyle{unsrtnat}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{bm}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\newcommand{\vect}[1]{\boldsymbol{\mathbf{#1}}}

\title{Skin Cancer Detection and Tracking\\using Data Synthesis and Deep Learning}

% The \author macro works with any number of authors. There are two
% commands used to separate the names and addresses of multiple
% authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to
% break the lines. Using \AND forces a line break at that point. So,
% if LaTeX puts 3 of 4 authors names on the first line, and the last
% on the second line, try using \AND instead of \And before the third
% author name.

\author{
  Yunzhu Li\thanks{indicate equal contribution.} \\
  Peking University\\
  \texttt{leo.liyunzhu@pku.edu.cn} \\
  %% examples of more authors
  \And
  Andre Esteva$^*$ \\
  Stanford University \\
  \texttt{esteva@cs.stanford.edu} \\
  \And
  Brett Kuprel \\
  Stanford University \\
  \texttt{brkuprel@gmail.com} \\
  \AND
  Rob Novoa \\
  Stanford University \\
  \texttt{ranovoa@gmail.com} \\
  \And
  Justin Ko \\
  Stanford University \\
  \texttt{jmko@stanford.edu} \\
  \And
  Sebastian Thrun \\
  Stanford University \\
  \texttt{thrun@stanford.edu}
}

\begin{document}
% \nipsfinalcopy is no longer used

\maketitle


% \begin{abstract}
%   Dense object detection and temporal tracking is needed across applications domains ranging from people-tracking to analysis of satellite imagery over time. The detection and tracking of malignant skin cancers and benign moles poses a particularly challenging problem due to the general uniformity of large skin patches, the fact that skin lesions vary little in their appearance, and the relatively small amount of data available. Here we introduce a novel data synthesis technique that merges images of individual skin lesions with full-body images and heavily augments them to generate significant amounts of data. We build a convolutional neural network (CNN) based system, trained on this synthetic data, and demonstrate superior performance to traditional detection and tracking techniques. Our system is intended for potential clinical use to augment the capabilities of healthcare providers. While domain-specific, we believe the methods invoked in this work will be useful in applying CNNs across domains that suffer from limited data availability.
% \end{abstract}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%  INTRODUCTION %%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{figure}[h!]
  \centering
  \includegraphics[width=0.7\textwidth]{pull.png}
  \caption{Key factors for skin cancer care include early detection and tracking over time.}
  \label{fig:pull}
\end{figure}

\section{Introduction}

Dermatology is a medical field which stands to be heavily augmented by the use of artificial intelligence techniques. 
Diseases are visually screened for, and many disease diagnoses are performed strictly with an in-clinic visual examination. 
Discerning between skin lesions is difficult - the difference between skin cancer (melanoma, carcinoma) and benign lesions (nevi, seborrheic keratosis) is minute (Figure~\ref{fig:pull}). 
With 5.4 million cases of skin cancer diagnosed each year in the United States alone, the need for quick and effective clinical screenings is rising ~\cite{rogers2015incidence,stern2010prevalence}.
Patients with skin cancer tend to be afflicted with many moles, and so one of the challenges in skin cancer screenings is identifying them amongst a myriad of benign lesions.
Another key element of these diagnoses is based on inspecting temporal changes in lesions - a fast changing lesion is more likely to be malignant. 
As such, patients and providers need tools to support this at scale.

Recent advances in detection and tracking using CNNs \cite{RCNN,R-FCN,WarpNet} has the potential to augment healthcare providers by (1) detecting points of malignancy, and (2) finding corresponding lesions across images, allowing them to be tracked temporally.
However, the primary challenge in using traditional detection techniques is working in a low-data regime without the availability of high volumes of annotated and labeled data - the largest existing open-source skin cancer dataset of photographic images is the Edinburgh Dermofit dataset, containing 1,300 biopsied images.
To overcome this we develop a domain-specific data synthesis technique which stitches small single-lesion images onto large body images.
Both the body images and the skin lesions images are heavily augmented with various techniques, and the lesions are blended onto the bodies using Poisson image editing \cite{PoissonImageEditing}. 

For large-scale lesion detection, we use this synthetic data to train a fully-convolutional CNN on the task of pixel-wise classification between three classes: background, benign lesion, malignant lesion, then post-processing to generate region proposals.
For image-to-image tracking, a network is trained (using image pairs containing altered lesion and pose positions) to output pixel-wise positional shift.

Our method demonstrates a working end-to-end CNN system capable of tackling two critical diagnostic tasks with superior performance to baseline techniques. It is trained with very little original data and thus the techniques demonstrated here can be easily transferred to other data-limited domains.




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%  DATA SYNTHESIS  %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Data Synthesis}
We create a domain-specific data augmentation technique for generating synthetic images from two low-data sources: high-quality lesion images (1,300 biopsy-proven cancers and moles), and body images (400 back, leg, and chest images) whose skin regions have been manually segmented.
We first generate images for detection and then further augment them for tracking.
These images are intended to mimic the real-world clinical case of patients exhibiting many lesions, some possibly malignant, with the need to track them over time. 

% \subsection{Detection Data}
Detection data is generated in two steps: (1) A blending position on the body image is chosen using local feature matching between a lesion image and the body image, (2) the lesion image is blended into the body image using Poisson image editing.
% \subsection{Tracking Data}
For tracking, data is generated by further augmenting detection images - that is, given a detection image, we create a pair of images from it.
The purpose here is to recreate temporal images (which will exhibit changes in lesion shape/size, as well as body changes) and to force the network to learn pixel-wise correspondence by focusing on the distortion in local texture information.

\begin{figure}%[h!]
  \centering
  \includegraphics[width=0.9\textwidth]{./blend.png}
  \caption{{\bf Data Synthesis.} (Left) Example biopsied skin lesion and raw body images. (Middle) Generated training images for detection and corresponding label masks. Red areas represent blended malignant lesions, yellow areas represent benign lesions. (Right) Generated training images for tracking, along with a few example pixel-wise correspondences.}
  \label{fig:poisson}
\end{figure}

% \begin{figure*}[t!]
%     \centering
%     \begin{subfigure}{\textwidth}
%         \centering
%         \includegraphics[width=\textwidth]{./blend.pdf}{a}
%           \caption{{\bf Data Synthesis.} Skin lesions are blended with raw body images to generate detection and tracking data. (Left) Example biopsied skin lesion and raw body images. Top diagrams show the lesion segmentation mask and the gradient field along with semantic regions used to calculate blending locations. (Middle) Generated training images for detection and corresponding label masks. Red areas represent blended malignant lesions, yellow areas represent blended benign lesions. (Right) Generated training images for tracking, along with a few example pixel-wise correspondences.}
%           \label{fig:poisson}
%     \end{subfigure}%
%     ~
%     \begin{subfigure}{\textwidth}
%         \centering
%         \includegraphics[width=\textwidth]{system.png}{b}
%           \caption{{\bf Detection and Tracking System.} The network is trained on synthetic data and tested on real data. Top row shows the detection pipeline, bottom row shows tracking. The detection network is composed of a convolution section followed by a deconvolution section, with skip-link connections between non-adjacent layers. The network outputs per-pixel labels over two malignant, two benign, and one background class. The tracking network takes the convolutional component of the detection network, and splits it up into a smaller convolutional part, and an atrous convolutional part. The two tracking images are each fed through the network and merged by a subtraction before the per-pixel shift prediction.
%           }
%           \label{fig:system}
%     \end{subfigure}
%     \caption{Caption place holder}
% \end{figure*}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%  SYSTEM PIPELINE %%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{System Pipeline}

% System figure
\begin{figure*}[h!]
  \centering
  \includegraphics[width=0.9\textwidth]{system.png}
  \caption{{\bf Detection and Tracking System.} The network is trained on synthetic data and tested on real data. The detection network is composed of a conv section followed by a deconv section, with skip-link connections. In the top right we show the raw prediction heat map and the detection result after post processing. The tracking network takes the conv component of the detection network, and splits it up into a smaller conv part, and an atrous conv part. The two tracking images are each fed through the network and merged by a subtraction before the per-pixel shift prediction.
  }
  \label{fig:system}
\end{figure*}

Our system is composed of two parts: the first detects malignant and benign skin lesions, the second tracks them across images.
The detection network is trained with the synthetic images (skin lesion + body images) described in the previous section, using pixel-wise labels.
Once the detection network is trained to convergence, its weights are used to initialize the tracking network. 
This network is then trained on image-pairs formed from the detection data.

% \subsection{Detection System}
The detection component is intended to highlight to a clinician the potentially malignant lesions on a given input image.
Providers are often faced with body regions containing a multitide of lesions and discerning malignancy is a challenging task. 
Our system feedforwards an input image through the CNN, outputs a pixel-wise heatmap over the five classes of interest, and then uses post-processing techniques to make the heatmap more human-interpretable. 
Examples of raw prediction results and post-processed images are shown in Figure~\ref{fig:raw_results}.

\begin{figure*}[b!]
  \centering
  \includegraphics[width=0.9\textwidth]{./qualitative.png}
  \caption{{\bf Qualitative Results.} Left: Detection results, Right: Tracking results. Four examples from our detection pipeline, compared to a baseline sliding-window classifier technique. Two examples from our tracking pipeline, compared to SIFT-Flow and DSP baselines, using $\alpha=0.05$.}
  \label{fig:raw_results}
\end{figure*}

% \subsection{Tracking System}

The tracking component of our system is intended to find pixel-wise correspondence between two images of the same body part, in order to track lesions over time. Shown in Figure~\ref{fig:system}, the tracking network is an adaptation of the detection network. During the feedforward pass the two images are fed through the net, which then outputs a 2D vector field of correspondences.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%  EXPERIMENTS AND RESULTS %%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Experiments and Results}

\begin{figure*}%[h!]
  \centering
  \includegraphics[width=0.3\textwidth]{./roc_detection.pdf}
  \includegraphics[width=0.3\textwidth]{./roc_tracking.pdf}
  \caption{{\bf Quantitative Results.} Detection results (left): ROC curve comparing our technique against a baseline sliding window method and two non-expert humans. Recall rate is shown in the parenthesis of the legend. Tracking Results (right): mean percentage of correct keypoints (PCK) as a function of $\alpha = p/L$, where $p$ is the number of pixels, and $L$ is the diagonal length of the image.}
  \label{fig:quantitative_results}
\end{figure*}

% \subsection{Detection}

Example results for detection are shown in the first three rows of Figure~\ref{fig:raw_results}, whose columns show examples as they are processed from input images to raw network output to post-processed results. 
Our method is compared to a baseline sliding window method (bottom row of Figure~\ref{fig:raw_results}).
For quantitative comparison, we test detection using 108 clinical images, and include results of two non-expert humans trained on Edinburgh Dermofit dataset to detect malignant lesions.
The generated receiver-operator curve shown in the left part of Figure~\ref{fig:quantitative_results}.

% \subsection{Tracking}
For tracking, we compare our results to two baseline techniques: SIFTFlow, and Deformable Spatial Pyramids (DSP) \cite{SIFTFlow,DeformableSpatialPyramid}.
Example image results are shown in Figure~\ref{fig:raw_results}, where green lines show correctly predicted correspondences, and red lines show incorrect predictions. 
For quantitative comparison, we evaluate tracking accuracy using the percentage-of-correct-keypoints (PCK) metric. Our test set is composed of 260 pairs of correspondence labels manually annotated on temporal image pairs. These image pairs vary in pose, background, distance, viewpoint, and illumination condition.
Results are plotted in Figure~\ref{fig:quantitative_results}, in comparison to both baselines.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%  CONCLUSION %%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Conclusion}

Here we show large-scale detection and tracking of skin lesions across images using FCN in a low-data regime using domain-specific data augmentation.
In the absence of large amounts of labeled and annotated data, we generate high volumes of synthetic data using 1,300 biopsy-proven clinical images of skin lesions and 400 body images. 
Skin lesion images are blended onto body images, heavily augmented with a variety of techniques, and used to train a detection network. 
We demonstrate human-interpretable detection with this method, and demonstrate superior performance over baseline.
We then further augment the data and generate image pairs with pixel-wise correspondence between them, and use this to train a tracking network whose architecture is partially composed of the detection network and initialized with its trained weights, outperforming both SIFTFlow and DSP. 
The networks are trained on synthetic data and tested on real-world data.

AI systems of this sort have the potential to improve the way healthcare is practiced, which may extend outside of the clinic.
Algorithms such as these could aid providers at spotting suspicious lesions amongst benign ones, and at observing temporal changes in lesions that may signify malignancies. 

\small
\bibliography{ref}


\end{document}

main.bbl
\begin{thebibliography}{8}
\providecommand{\natexlab}[1]{#1}
\providecommand{\url}[1]{\texttt{#1}}
\expandafter\ifx\csname urlstyle\endcsname\relax
  \providecommand{\doi}[1]{doi: #1}\else
  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi

\bibitem[Rogers et~al.(2015)Rogers, Weinstock, Feldman, and
  Coldiron]{rogers2015incidence}
Howard~W Rogers, Martin~A Weinstock, Steven~R Feldman, and Brett~M Coldiron.
\newblock Incidence estimate of nonmelanoma skin cancer (keratinocyte
  carcinomas) in the us population, 2012.
\newblock \emph{JAMA dermatology}, 151\penalty0 (10):\penalty0 1081--1086,
  2015.

\bibitem[Stern(2010)]{stern2010prevalence}
Robert~S Stern.
\newblock Prevalence of a history of skin cancer in 2007: results of an
  incidence-based model.
\newblock \emph{Archives of dermatology}, 146\penalty0 (3):\penalty0 279--282,
  2010.

\bibitem[Girshick et~al.(2014)Girshick, Donahue, Darrell, and Malik]{RCNN}
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.
\newblock Rich feature hierarchies for accurate object detection and semantic
  segmentation.
\newblock In \emph{Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition ({CVPR})}, 2014.

\bibitem[Jifeng~Dai(2016)]{R-FCN}
Kaiming He Jian~Sun Jifeng~Dai, Yi~Li.
\newblock {R-FCN}: Object detection via region-based fully convolutional
  networks.
\newblock \emph{arXiv preprint arXiv:1605.06409}, 2016.

\bibitem[Kanazawa et~al.(2016)Kanazawa, Jacobs, and Chandraker]{WarpNet}
Angjoo Kanazawa, David~W. Jacobs, and Manmohan Chandraker.
\newblock Warpnet: Weakly supervised matching for single-view reconstruction.
\newblock In \emph{The IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR)}, June 2016.

\bibitem[P{\'e}rez et~al.(2003)P{\'e}rez, Gangnet, and
  Blake]{PoissonImageEditing}
Patrick P{\'e}rez, Michel Gangnet, and Andrew Blake.
\newblock Poisson image editing.
\newblock \emph{ACM Trans. Graph.}, 22\penalty0 (3):\penalty0 313--318, July
  2003.
\newblock ISSN 0730-0301.
\newblock \doi{10.1145/882262.882269}.
\newblock URL \url{http://doi.acm.org/10.1145/882262.882269}.

\bibitem[Liu et~al.(2008)Liu, Yuen, Torralba, Sivic, and Freeman]{SIFTFlow}
Ce~Liu, Jenny Yuen, Antonio Torralba, Josef Sivic, and William~T Freeman.
\newblock Sift flow: Dense correspondence across different scenes.
\newblock In \emph{European conference on computer vision}, pages 28--42.
  Springer, 2008.

\bibitem[Kim et~al.(2013)Kim, Liu, Sha, and Grauman]{DeformableSpatialPyramid}
Jaechul Kim, Ce~Liu, Fei Sha, and Kristen Grauman.
\newblock Deformable spatial pyramid matching for fast dense correspondences.
\newblock In \emph{The IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR)}, June 2013.

\end{thebibliography}
