Mediastinal lymph nodes segmentation using 3D convolutional neural network ensembles and anatomical priors guiding
Purpose: As lung cancer evolves, the presence of enlarged and potentially malignant lymph nodes must be assessed to properly estimate disease progression and select the best treatment strategy. Following the clinical guidelines, estimation of short-axis diameter and mediastinum station are paramount for correct diagnosis. A method for accurate and automatic segmentation is hence decisive for quantitatively describing lymph nodes.\\ Methods: In this study, the use of 3D convolutional neural networks, either through slab-wise schemes or the leveraging of downsampled entire volumes, is investigated. Furthermore, the potential impact from simple ensemble strategies is considered. As lymph nodes have similar attenuation values to nearby anatomical structures, we suggest using the knowledge of other organs as prior information to guide the segmentation task.\\ Results: To assess the segmentation and instance detection performances, a 5-fold cross-validation strategy was followed over a dataset of $120$ contrast-enhanced CT volumes. For the $1\,178$ lymph nodes with a short-axis diameter $\geq10$\,mm, our best performing approach reached a patient-wise recall of $92$\%, a false positive per patient ratio of 5, and a segmentation overlap of $80.5$\%. The method performs similarly well across all stations.\\ Conclusion: Fusing a slab-wise and a full volume approach within an ensemble scheme generated the best performances. The anatomical priors guiding strategy is promising, yet a larger set than four organs appears needed to generate an optimal benefit. A larger dataset is also mandatory, given the wide range of expressions a lymph node can exhibit (i.e., shape, location, and attenuation), and contrast uptake variations.
article.tex
\documentclass{article}


\usepackage{arxiv}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{lipsum}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{adjustbox}
\usepackage{multirow}
\usepackage{multicol}
\usepackage{xcolor}
\usepackage{authblk}
\graphicspath{ {./} }


\title{Mediastinal lymph nodes segmentation using 3D convolutional neural network ensembles and anatomical priors guiding}


\author[1,2]{David Bouget}
\author[1]{Andr{\'e} Pedersen}
\author[1]{Johanna Vanel}
\author[3]{Haakon O. Leira}
\author[1]{Thomas Lang{\o}}
\affil[1]{Department of Medical Technology\\
  SINTEF\\
  Trondheim, Norway}
\affil[2]{Center for Innovative Ultrasound Solutions, NTNU, Trondheim, Norway}
\affil[3]{ Department of Thoracic Medicine\\
  St. Olavs hospital\\
  Trondheim, Norway }

\begin{document}
\maketitle
\begin{abstract}
\textbf{Purpose:} As lung cancer evolves, the presence of enlarged and potentially malignant lymph nodes must be assessed to properly estimate disease progression and select the best treatment strategy. Following the clinical guidelines, estimation of short-axis diameter and mediastinum station are paramount for correct diagnosis. A method for accurate and automatic segmentation is hence decisive for quantitatively describing lymph nodes.\\
\textbf{Methods:} In this study, the use of 3D convolutional neural networks, either through slab-wise schemes or the leveraging of downsampled entire volumes, is investigated. Furthermore, the potential impact from simple ensemble strategies is considered. As lymph nodes have similar attenuation values to nearby anatomical structures, we suggest using the knowledge of other organs as prior information to guide the segmentation task.\\
\textbf{Results:} To assess the segmentation and instance detection performances, a 5-fold cross-validation strategy was followed over a dataset of $120$ contrast-enhanced CT volumes. For the $1\,178$ lymph nodes with a short-axis diameter $\geq10$\,mm, our best performing approach reached a patient-wise recall of $92$\%, a false positive per patient ratio of 5, and a segmentation overlap of $80.5$\%. The method performs similarly well across all stations.\\
\textbf{Conclusion:} Fusing a slab-wise and a full volume approach within an ensemble scheme generated the best performances. The anatomical priors guiding strategy is promising, yet a larger set than four organs appears needed to generate an optimal benefit. A larger dataset is also mandatory, given the wide range of expressions a lymph node can exhibit (i.e., shape, location, and attenuation), and contrast uptake variations.
\end{abstract}

% keywords can be removed
\keywords{3D segmentation, Ensemble, Attention U-Net, Mediastinal lymph nodes, CT}


\section{Introduction}
\label{intro}
Lung cancer manifests itself through the development of malignant tumors characterized by uncontrolled cell growth in tissues of the lung. As cancer evolves, its growth can spread beyond the lung and reach nearby anatomical structures such as lymph nodes, causing them to grow in size~\cite{falk2010}. Lymph nodes are routinely assessed by clinicians to monitor disease progression, establish cancer diagnosis, or simply evaluate the effect of therapeutics given their propensity to enlarge under the effect of many pathologies. As defined by the Response Evaluation Criteria In Solid Tumors (RECIST) guidelines~\cite{eisenhauer2009new}, a lymph node with a short-axis diameter of at least $10$\,mm in an axial CT slice is likely to be malignant and represent a clinical interest~\cite{schwartz2009evaluation}.
Accurate clinical or pretreatment stage classification of lung cancer leads to optimal patient outcomes and improved prognostication. Regional lymph node maps are mandatory to facilitate consistent and reproducible lymph node designations, and are regularly issued by the International Association for the Study of Lung Cancer (IASLC)~\cite{el2014international}. The regional map defines fourteen different general anatomic locations, also called stations, precisely described by a set of guidelines articulated around neighbouring anatomical structures in the mediastinum (e.g., blood vessels, airways, or ligaments). In practice, a lymph node is assigned its station by an expert radiologist, according to its relative position with respect to nearby anatomical structures.
In lung cancer diagnosis, a chest contrast-enhanced Computed Tomography (CT) scan is most frequently favored, and represents the gold-standard modality. In absence of distant metastasis, enlarged lymph nodes identified on chest CT demand a verification procedure, either through endobronchial ultrasound (EBUS)~\cite{sorger2017multimodal} or mediastinoscopy, to ascertain the severity and aggressiveness of the cancer. However, the manual segmentation of lymph nodes in the mediastinal area is tedious, highly time-consuming, and requires trained experts. The process is inherently subject to intra/inter-observer variability depending on the time allocated to perform the task, level of concentration, and quality of the CT scan~\cite{mcerlean2013intra}. In addition, challenges arise from the relatively similar attenuation between lymph nodes and surrounding structures (e.g., esophagus, azygos vein, or other vessels), especially impactful when poor contrast enhancement is exhibited. Last but not least, lymph nodes manifest themselves through extensive variations in shape, size, texture, and location.
An automatic method is therefore of high importance for facilitating the tasks of lymph node segmentation and standardized measurements computation (e.g., short-axis diameter, station) in order to assist the clinical team in making the best cancer staging.


\setlength{\parindent}{5ex} Advances in machine learning, and more specifically deep learning, has boosted the performance in image segmentation, and fully convolutional neural networks~\cite{long2015fully} have been widely accepted for medical image segmentation~\cite{ronneberger2015u,zhou2017fixed}. While deep learning-based methods thrive on everyday-life data, access to sufficiently large and annotated training datasets in the medical field represents a known bottleneck. In recent years, medical image segmentation, from CT volumes amongst other modalities, has known tremendous progress, driven by subsequent challenges where annotated datasets were made publicly available to foster research and challenge medical communities. For instance, the segmentation of moderately large organs (e.g., liver, pancreas, or kidneys) from abdominal CT volumes was initially investigated in multiple challenges~\cite{kavur2020chaos,simpson2019large}. Such organs present the advantage of being easy to delineate, without the required assistance of an expert radiologist, to create trustworthy datasets. Initially, most current deep learning architectures were designed to operate using a 2D input, hence requiring CT volumes to be segmented one sectional image (i.e., slice) at a time~\cite{vesal20192d,wang2019abdominal}. Yet in practice, a radiologist would scroll through the CT volume and across all views to properly identify the full extent of a given anatomical structure. Naturally, anatomical borders can be only discernible from subtle change in texture or shape, not always visible in consecutive 2D slices.
To overcome such limitations, attempts have been made to analyze all three views at the same time in a 2.5D fashion~\cite{zhuang2016multi}, or by using multiple 2D patches around the segmentation target~\cite{setio2016pulmonary}. Fortunately, with the increasing capacity of GPUs, and the ability for neural network architectures to seamlessly adapt to different dimensions, studies over 3D medical images emerged~\cite{cciccek20163d}. Attempts to leverage full resolution CT volumes are still held back by their huge memory footprint. The rising bottlenecks comprise a high model complexity (i.e., number of parameters), longer training times, and overfitting issues. Nevertheless, access to local 3D context can be gained by processing the raw volume in a slab-wise or block-wise fashion~\cite{kamnitsas2016deepmedic}.
For the joint tasks of segmentation and instance detection, popular approaches have been widely used over 2D images~\cite{he2017mask,redmon2018yolov3}. While such architectures are powerful and could translate well to the 3D domain, setting up the region proposal layer is too memory-expensive and training time would be a challenge. 


In the literature, mediastinal lymph nodes have predominantly been studied for detection purposes and on occasion for segmentation, most often leveraging only a contrast-enhanced CT volume. In their initial work, Oda et al. employed standard machine learning technique to perform lymph node detection whereby hand-crafted features were extracted using a Hessian-based strategy~\cite{oda2017hessian}. In a follow-up study, a two-step detection algorithm was proposed, based on an intensity targeted radial structure tensor and blob-like structure enhancement filters~\cite{oda2017automated}. Similarly, Paing et al. ~\cite{paing2019automatic} proposed a pipeline mixing traditional image features extraction (i.e., threshold, watershed, and hessian eigenvalues) fed to a classification 3D neural network separating lymph node candidates from false lesions. In mediastinal and abdominal contexts, Nogues et al. focused on the segmentation of lymph node clusters~\cite{nogues2016automatic}. The cope with reduced intensity and texture contrast amongst collocated lymph nodes, 2D holistically-nested neural networks were proposed as a solution to perform embedded edge detection. Structured optimization techniques were then promoted to refine the produced imprecise segmentation, such as conditional random fields and graph cuts. Roth et al.~\cite{roth2014new} presented a two-stage pipeline to perform detection of all the potentially malignant lymph nodes. All voxels were first classified as to either belonging to the lymph node class or the background by use of blobness and circular transforms. Fed by the results of the first stage, a convolutional neural network, trained in a 2.5D fashion, produced the final set of lymph node candidates.
Overall, the proposed methods were partly or fully built upon ad-hoc components or required strongly hand-crafted features, which represents a clear limitation given the impossibility to assume fixed intensity thresholds from input volumes with varying quality. In addition, prior assumptions over the shape of a lymph node were made (i.e., roundish blob) and are often incorrect. In consecutive works, Liu et al. addressed the topic of mediastinal lymph node detection and station mapping using the previously described two-stage pipeline~\cite{liu2014mediastinal,liu2016mediastinal}. Eight anatomical structures were additionally segmented to help in the station mapping task. Each lymph node candidate was assigned a station based on its centroid location w.r.t. surrounding structures and following the IASLC guidelines. Five stages and more than half an hour were necessary to produce the final segmentation and instance detection results. The use of hand-crafted features and multiple steps also limit the method's ability to generalize and deployment in practice. Fully end-to-end approaches appears more adequate to address both limitations at once. To that end, Oda et al.~\cite{oda2018dense} proposed a method to perform lymph node detection and segmentation using a 3D U-Net. To improve their performances and reduce data imbalance issues, they also proposed to include four anatomical structures (i.e., lungs, airways, aortic arch and branches, and pulmonary arteries). While a high true positive detection rate was reached, the reported false positive per patient rate of 17 appeared quite high. Including up to fourteen anatomical structures, in addition to the lymph nodes, Bouget et al. proposed a pipeline operating in 2D~\cite{bouget2019semantic}. The suggested approach combined the pixel-wise segmentation capability of a U-Net and improved instance detection capability from Mask R-CNN. Validated only on a dataset of $15$ patients, a recall of $75$\% was reached for $9$ false positives per patient on average, but the lack of global information was detrimental to obtain competitive results.
Facing the issue of limited access to ground truth, Li et al.~\cite{li2020deep} devised a weakly supervised method generating bounding boxes and pixel-wise pseudo-mask from the RECIST annotations. A U-Net architecture was used to produce the initial pixel-wise segmentation and a deep reinforcement-based component was coupled for optimization. State-of-the-art performance was attained over thoracoabdominal lymph nodes, reaching a Dice score of $77$\%.
Many previous studies performed their experiments using the only open-source dataset, first introduced by Roth et al.~\cite{roth2014new}. The dataset contains $90$ CT volumes featuring mediastinal lymph nodes, together with manual annotations provided for most lymph nodes with a short-axis diameter larger than $10$\,mm. However, the annotations are often sparse, and every smaller but visible lymph node was left unsegmented, making it suboptimal to use straight off the shelf for fully supervised pixel-wise segmentation purposes.


In another line of work, a few studies have focused on leveraging both contrast-enhanced CT and PET/CT modalities concurrently. Only the malignant lymph nodes are exhibiting uptake in the PET volume leaving the benign ones unnoticeable. For simple pixel-wise semantic segmentation, Xu et al. proposed to use the DeepLabv3+ architecture to benefit from atrous spatial pyramid pooling operating at various grid scales, improving boundary segmentation~\cite{xu2020focal}. Focal loss was further studied to let the network focus on the difficulty-to-segment voxels and prevent overfitting on the other category of nonchallenging voxels. Zhu et al.~\cite{zhu2020lymph} proposed a multi-branch detection-by-segmentation network to perform segmentation and detection of malignant lymph nodes. An effective distance-based gating approach was developed in their proposed framework, replicating protocols conducted by oncologists in daily practice. Unfortunately, a PET/CT scan is usually acquired at a later stage during the diagnostic process, after cancer suspicion is raised from the analysis of the contrast-enhanced CT scan. As such, an optimal processing of this initial CT volume is mandatory before considering the PET/CT modality.


In this study, our contribution are the following: (i) the investigation of slab-wise, full volume, and ensemble strategies for semantic segmentation in 3D to benefit from local and global context, (ii) validation studies showing segmentation and detection performances with respect to lymph nodes' short-axis diameter and station providing insights on the impact of training data variability, (iii) the largest annotated mediastinal lymph nodes dataset with $120$ patients and close to $3\,000$ lymph nodes, partially available in open-access.


\section{Data}
\label{sec:dataset}
\begin{figure}[ht]
\centering
\includegraphics[scale=0.79]{image1.png}
\caption{Illustrations of the manual annotations over the dataset. For each pair, the left image represents the raw CT clipped to $[-150;500]$\,HU, and the right image represents the same CT slice with the manually annotated lymph nodes overlaid in color. From top to bottom, and left to right, each example features lymph nodes in different stations.}
\label{fig:dataset-illu}
\end{figure}

In this work, a dataset of $120$ contrast-enhanced CT volumes was assembled, featuring lung cancer patients exhibiting malignant lymph nodes and coming from two different sources. First, the $89$ patients with mediastinal lymph nodes from the NIH open-source dataset~\cite{roth2014new} were gathered, which are publicly available for download~\footnote{https://wiki.cancerimagingarchive.net/display/Public/CT+Lymph+Nodes}. The remaining $31$ volumes were acquired on lung cancer patients at the Department of Thoracic Medicine, St. Olavs hospital, Trondheim University Hospital, Norway.

Overall, CT volume dimensions are covering $[487; 512]\times[441; 512]\times[56; 854]$\,voxels, and the voxel size ranges $[0.58; 0.97]\times[0.58; 0.97]\times[0.5; 5.0]$\,mm$^{3}$. An average CT volume is $[511\times511\times628]$ pixels with a spacing of $[0.79\times0.79\times0.99]$\,mm$^3$.
For the CT volumes acquired at the St. Olavs hospital, lymph nodes' manual annotations were performed by an expert thoracic radiologist. Regarding the CT volumes from the NIH dataset, the available annotations were used as a starting point and manually refined by a medical trainee under the supervision of the expert. Following the RECIST criterion, malignancy for a lymph node is considered for a short-axis diameter larger than $10$\,mm. Using the \textit{regionprops} method from the Scikit-Image Python package, the short-axis diameter was computed for each annotated lymph node. A total of $2\,912$ lymph nodes are featured in our dataset, with $1\,178$ having a short-axis diameter larger than $10$\,mm, $767$ having a short-axis diameter in the range $[7, 10[$\,mm, and $967$ with a short-axis diameter smaller than $7$\,mm. A set of annotated lymph nodes from our dataset is illustrated in Fig~\ref{fig:dataset-illu}.


Following the IASLC guidelines, each lymph node was assigned its station by a medical trainee under supervision of the expert. The exhaustive list of stations' number and name is as follows: 1) Low cervical, supraclavicular, and sternal notch nodes, 2) Upper paratracheal, 3a) Prevascular, 3p) Retrotracheal, 4) Lower paratracheal, 5) Subaortic, 6) Para-aortic, 7) Subcarinal, 8) Paraesophageal below carina, 9) Pulmonary ligament, 10) Hilar, 11) Interlobar, 12) Lobar, 13) Segmental, and 14) Sub-segmental. Most stations can also be subdivided between left and right side, but in the rest of the paper we will refer to a station by its main number. Given the large variability in lymph nodes' expression (e.g., shape and size), overlaps across multiple stations are common and multiple station assignments were required. Occasionally, lymph nodes could not be directly mapped to the guidelines or were outside the mediastinum scope, and were thus left station-less (e.g., around the heart, below the base of the lungs, or at the base of the neck). Overall, $74$ lymph nodes were left unassigned, $1\,256$ were overlapping at least two stations and $379$ were overlapping at least three stations. In terms of volume, lymph nodes are ranging $[0.01, 234.72]$\,ml with a mean value of $1.98\pm6.81$\,ml. Lymph nodes statistics with respect to volume and primary station are illustrated in Fig.~\ref{fig:dataset-ln-stations}. The primary station distribution is represented to the left, with unassigned lymph node grouped in the NA category. A large imbalance can be noticed, especially between station 2 and 9, and it can be noted that on average lymph nodes are more present on the right side. In some cases, the decision could not be made regarding lateralization, represented as unspecified (in blue). Volume-wise, the distribution across primary stations is relatively homogeneous but the plot cannot be interpreted unbiasedly as often more than one station is overlapped by a lymph node, especially for the larger ones.

\begin{figure}[t]
\centering
\includegraphics[scale=0.55]{image2.png}~\includegraphics[scale=0.55]{image3.png}
\caption{Dataset statistics regarding the primary station distribution (to the left), and the volume distribution within each primary station (to the right). Each station is uniquely represented and the lateralization distribution is not directly shown. Lymph nodes not mapped to any IASLC station are regrouped under the NA category.}
\label{fig:dataset-ln-stations}
\end{figure}

\section{Methods}
First, a description of the neural network architectures and designs used in this study is provided in Section~\ref{subsec:architectures}. Then, the possibility to leverage anatomical priors to guide the network during training is introduced, together with our proposed ensemble strategy, in Section~\ref{subsec:anatomical-prior}. Finally, the different preprocessing steps and selected training strategies are detailed in Section~\ref{subsec:train-strats}. An overview of our proposed approach is given in Fig.~\ref{fig:overall-achitecture}.

\subsection{Architectures and designs}
\label{subsec:architectures}
For properly segmenting and detecting instances of potentially collocated lymph nodes, the most promising alternatives described in the literature are multi-task architectures the likes of Mask R-CNN~\cite{he2017mask} or YOLACT~\cite{bolya2019yolact}. Even though  extremely good performance was documented from studies carried out in the 2D domain, fewer studies were applied to the 3D domain as the transition presents many limitations ranging from memory overload, to complexity to generate enough sensible proposals.
When dealing with 3D medical volumes, too sizable to fit on GPU memory, two main lines of work can be identified: (i) splitting the volume in a slab-wise manner (SW) and (ii) using the full volume at a lower resolution (FV). In the slab-wise strategy, the objective is to benefit locally from a high-resolution while generating contextual features on some extent of global information, which is correlated to the slab size. In the full volume strategy, overall contextual features can be computed to model all spatial relationships between each visible anatomical structure. However, due to memory limitation and time constraint for training, keeping the initial resolution is not feasible yet and degrading the spatial resolution is needed as opposed to the slab-wise strategy.
Over the previous years, ample focus has been dedicated to improving 2D/3D pixel-wise segmentation performances over well-established backbone architecture such as ResNet~\cite{he2016deep} and U-Net~\cite{ronneberger2015u}, providing near radiologist-level performances on some medical image analysis tasks~\cite{bai2017human,liao2019evaluate}. The common pitfall for such encoder-decoder architectures is some extent of details' loss provoked by strided convolutions and pooling operations along the encoding path to progressively enlarge the field-of-view. The constant challenge lies in optimally using global and contextual information from high-level features and border information from low-level features to resolve small details~\cite{sang2020pcanet}.
To compensate for the loss of details in the encoding path, previous studies have non-exhaustively covered multi-scale investigations employing the input volume at down-sampled scales in each encoder block~\cite{abraham2019novel}, enlarging the receptive fields using atrous convolutions and pyramid spatial pooling~\cite{chen2017rethinking}, or by crafting multi-scale feature maps encoding jointly low-level and high-level semantic information in a powerful representation~\cite{sinha2020multi}. Similarly, instead of computing the loss simply from the last decoder step intermediate feature maps generated at each level of the architecture can be leveraged. Feature maps from hidden layers of a deep network can serve as a proxy to improve the overall segmentation quality and sensitivity of the model, while alleviating the problem of vanishing gradients~\cite{lee2015deeply}.
In a different line of work, attention mechanisms are able to identify salient image regions and amplify their influence while filtering away irrelevant information from other regions, making the prediction more contextualised~\cite{jetley2018learn}. Attention is optimally coupled to each level during the decoding path and can be designed to capture features' dependencies spatially, channel-wise, or across any other dimension~\cite{fu2019dual}. Interestingly, all those suggested concepts can be seamlessly integrated into the current CNN architectures considered as backbones and enable complete end-to-end training.

\label{sec:methods}
\begin{figure}[!t]
\centering
\includegraphics[scale=0.5]{image4.png}
\caption{Overall representation of the proposed approach, where a slab-wise and a full volume approach are fused in an ensemble fashion, with the possibility to include anatomical priors.}
\label{fig:overall-achitecture}
\end{figure}

In this work, given the inherent 3D nature of the various anatomical structures in the mediastinal area, we chose to focus on improving the lymph nodes' pixel-wise segmentation using 3D U-Net~\cite{cciccek20163d} as backbone architecture (denoted as UNet). Our UNet design has been set to seven levels and $[8, 16, 32, 64, 128, 256, 256]$ as filter sizes for each level respectively. For studying the impact of concepts like deep supervision or attention, we chose the Attention-Gated U-Net (AGUNet) and Dual-Attention Guided U-Net (DAGUNet) architectures as previously introduced~\cite{bouget2021meningioma}. For those, the design was set to five levels and $[32, 64, 128, 256, 512]$ as filter sizes for each level respectively.

\subsection{Anatomical knowledge priors and model ensembles}
\label{subsec:anatomical-prior}
In the mediastinal area, many anatomical structures have attenuation values in the same range as the lymph nodes (e.g., esophagus, azygos vein) often without clear in-between boundaries. To assist the model in better differentiating between lymph nodes and surrounding similar-looking structures, anatomical knowledge priors can be injected to serve as prior knowledge during training. To that end, each training sample can be built as a combination of the raw CT positioned in channel 0, and a binary mask containing anatomical priors placed in channel 1. With this approach, the no-go zones defined in the anatomical priors' mask should be identified to prevent the model generating high probabilities, hence decreasing the false positive ratio.

Multiple models operating on different input shapes or focusing on different aspects during training can be ensembled to generate a better consensus and hence improve the final prediction map~\cite{feng2020brain}. Global context and local refinement can virtually be obtained separately at the cost of longer training and inference time, and higher model complexity.
Regarding the task of lymph nodes' segmentation, the two main strategies considered (i.e., slab-wise and full volume) operate on different domains and present each inherent strengths and weaknesses. Since training a model using the whole CT at a high resolution is not feasible, performing ensembling appears to be a competitive alternative solution. In this study, we opted for a straightforward approach whereby a maximum operator is applied pixel-wise over the probability maps resulting from a model trained following each of the two main strategies. By doing so,
the newly created probability map is biased toward a better recall at the detriment of the precision since more voxels will have a higher probability as belonging to the lymph node class.

\subsection{Training strategies}
\label{subsec:train-strats}
To prepare the training samples, all CT volumes were preprocessed identically using the following series of steps: (i) resampling to an isotropic spacing of $1$\,mm$^{3}$ using spline interpolation order 1 from NiBabel~\footnote{https://github.com/nipy/nibabel}, (ii) lung-cropping using a pre-trained network~\cite{hofmanninger2020automatic} in order to generate the tightest bounding box around the mediastinal area, (iii) resizing to the network's input resolution using spline interpolation order 1, and (iv) intensity clipping to the range $[-250, 500]$\,HU followed by normalizing to the range $[0, 1]$.


For the slab-wise investigations performed in this study, we chose to wield slabs of 32 slices and 64 slices along the z-axis denoted as SW32 and SW64 respectively, for an axial resolution of $256\times192$\,pixels. In order to generate the collection of training samples, a stride parameter of 8 was employed whereby two consecutive slabs would share 24 slices and 56 slices respectively. The value was empirically chosen as a good trade-off by which models can be trained in a decent amount of time, generalize well, and circumvent overfitting hurdles.
Regarding the full volume approach, a drastic down-sampling was imperative given the GPU memory limitations, therefore a new resolution of $128\times128\times144$\,voxels was chosen.
Investigations performed using anatomical priors guiding were performed solely with the full volume approach, hence with a final training sample dimension of $128\times128\times144\times2$\,voxels. The four following structures were used as part of the anatomical priors: esophagus, azygos vein, subclavian arteries, and brachiocephalic veins.


For the data augmentation strategy, the following transforms were applied to each input sample with a probability of $50$\%: horizontal and vertical flipping, random rotation in the range $[-20^{\circ}, 20^{\circ}]$, translation up to 10\% of the axis dimension, zoom between $[80, 120]\%$ in the axial plane.
Given the large variability in lymph nodes' shape and location, and the relatively limited total number of patients, the training samples were randomly assigned to their fold at a patient-level without any other consideration. All models were trained from scratch using the Adam optimizer with an initial learning rate of $10^{-3}$, and training was stopped after $30$ consecutive epochs without validation loss improvement. The main loss function used was the class-average Dice, excluding the background, and batches of size 8 were used for slab-wise training, while batches of size 1 with 32 accumulated gradient steps were used for full volume training. The concept of accumulated gradient enables training with larger batch sizes, whereby N batches are run sequentially using the same model weights for calculating the gradients. When the N steps are performed, the model weights are properly updated using the accumulated sum of gradients.

\section{Validation studies}
\label{sec:validation}
In this work, we aim at assessing the lymph nodes' pixel-wise segmentation and partial instance detection performances, leaving aside the task of disambiguation between collocated instances. The dataset was split randomly into 5 folds at the patient level and a 5-fold cross-validation performed whereby three folds were used for training, one for validation, and one for testing. As described in Section~\ref{subsec:gt-preproc}, the initial ground truth was slightly refined to adjust for the inability to distinguish between collocated lymph nodes. The selected measurements and metrics used are then introduced in Section~\ref{subsec:meas-met} while the validation studies are detailed in Section~\ref{subsec:val-studies}.

\subsection{Ground truth preprocessing}
\label{subsec:gt-preproc}
In order to assess instance detection performances (e.g., recall, precision) over results from a pure pixel-wise segmentation neural network, some adjustments must be performed.
From the resulting probability map, and after thresholding using the optimal value, a connected components approach is operated in order to generate lymph node instances. However, collocated lymph nodes would only produce one instance using such an approach. In addition, the same behaviour is to be expected for close-by lymph nodes, given some noisiness in the final binary mask.
To reflect the inability of our method to optimally perform instance segmentation and reduce the bias in reported performances, we applied the same connected components method on the original ground truth. Each new lymph node cluster was assigned with all primary stations for each individual element composing the cluster. An illustration is provided in Fig.~\ref{fig:cc-gt-illu}, where two clusters of collocated lymph nodes (circled in red) are visible.

\begin{figure}[t]
\centering
\includegraphics[scale=0.94]{image5.png}
\caption{Illustration of the preprocessing performed over the ground truth annotation to prepare for instance detection performance assessment. The raw CT slice is shown to the left, the original ground truth with 7 lymph nodes is visible in the middle, and the final ground truth with 3 lymph nodes is presented to the right. The two transformed lymph node clusters are circled in red.}
\label{fig:cc-gt-illu}
\end{figure}

\subsection{Measurements and metrics}
\label{subsec:meas-met}
To assess the segmentation performance, the Dice score is computed between the ground truth and a binary representation of the probability map generated by a trained model. The binary representation is computed for ten different equally-spaced probability thresholds (PT), in the range $[0, 1]$.
After identification of the best PT for each model, the instance detection performance is assessed in turn. A connected components approach, used over the resulting binary map, coupled to a pairing strategy is executed to identify matches between a ground truth and a detection mask, based on the Dice score computation.

The Dice score, reported in \%, is used to assess the quality of the pixel-wise segmentation at a patient level. To refine and highlight the reported segmentation performance at a lymph node level, the Dice score over true positive instances is further computed (noted Dice-TP). Finally, discrepancies are expected between paired lymph node instances from the use of a connected components approach over noisy masks. To prevent the results from being heavily penalized by such shortcomings, we propose to report in complement the total percentage of the ground truth to have been correctly segmented at a lymph node level (noted GT-perc).


To report instance detection performance, the recall as a global metric across all patients is supplied, as well as a per-patient recall (noted Recall-PW) to take into account the high variability in the amount of lymph nodes from patient to patient. Regarding precision, the number of false positives per patient (FPPP) is mentioned.


Lastly, the pure inference speed and the total elapsed time required to generate predictions for a new patient are reported (in s), obtained with both CPU or GPU support. The operations required to prepare the data to be sent through the network, to initialize the environment, to load the trained model, and to reconstruct the probability map in the referential space of the original volume are accounted for. The experiment has been repeated five consecutive times over the same CT volume, using a representative high-resolution case of $512\times512\times767$\,voxels with $0.68\times0.68\times0.5$\,mm$^{3}$ spacing.
All measurements are reported with mean and standard deviation.

\subsection{Studies}
\label{subsec:val-studies}
Experiments using various combinations of architectures and training strategies (as introduced in Section~\ref{sec:methods}) were carried out, and the name given to each experiment is a descriptive concatenation. Architectures to choose from are: regular U-Net (UNet), attention-gated U-Net (AGUNet), and dual attention guided U-Net (DAGUNet). Training strategies to choose from are: slab-wise with 32 slices (SW32), slab-wise with 64 slices (SW64), full volume (FV), and using anatomical priors guiding (APG). Experiments using ensembles are specifically mentioned.

\textit{Slab-wise performance analysis:} On average, twelve days were necessary to train one model in a slab-wise fashion. Under such circumstances, comparing all slab-wise methods using a 5-fold cross-validation approach was realistically unachievable. The objective of this first study is to show limited performance variability across the five different folds using the UNet-SW32 approach. All models trained in a slab-wise fashion are then compared over the first fold only to identify the best performing approach.

\textit{Overall performance comparison:} Averaged over the 5 folds, a comparative analysis is performed between the different methods considered and across the whole set of proposed metrics. In this study, all lymph nodes featured in our dataset are considered without restrictions over the short-axis diameter or primary station. A total of $2\,122$ lymph nodes or lymph node clusters, after ground truth preprocessing, are taken into account.

\textit{Performance analysis over lymph nodes' characteristics:} For the best performing model identified in the previous validation study, an in-depth analysis is carried out with respect to the short-axis diameter and primary station.
Three divisions are used to highlight the performance with regard to the short-axis diameter, around the thresholds of $7$\,mm and $10$\,mm.
Given our choice of preprocessing steps to generate the training samples, many lymph nodes featured in station 1 (i.e., above or at the top of the lungs) were never seen by the network and hence could not be segmented. As such, two main groups are considered: (i) all stations even for the NA category (noted all stations), and (ii) all stations except 1 and NA (noted relevant stations).


\textit{Ground truth quality assessment over a benchmark subset:} Out of the $31$ CT volumes acquired at the St. Olavs hospital and featured in our dataset, $15$ have been part of previous studies and are publicly available~\cite{reynisson2015airway}. To assess the quality of the ground truth, a second expert with a background as a thorax radiologist was asked to look over the segmentation and assigned stations for this subset. To judge the inter-rater variability regarding station assignment, and given that lymph nodes can overlap multiple stations, we defined three grades: (i) perfect if both annotators were in agreement regarding the primary station, (ii) good if annotators agreed on the multiple stations overlapped but disagreed on the primary station, and (iii) bad if the second expert assigned a different and unrelated primary station.\\
In addition to the ground truth made publicly available, segmentation and instance detection performances obtained over the $15$ patients are separately reported for benchmark purposes.

\section{Results}
\label{sec:results}

Models were trained using an HP desktop: Intel Xeon @3.70 GHz, 62.5 GiB of RAM, NVIDIA Quadro P5000 (16GB), and a regular hard drive. Implementation was done in Python 3.6 using \texttt{TensorFlow} v1.13.1, Cuda 10.0, and the Imgaug Python library for the data augmentation methods~\cite{imgaug}.
Trained models, inference code, and ground truth files are available at~\url{https://github.com/dbouget/ct_mediastinal_structures_segmentation}.

\subsection{Slab-wise performance analysis}
For each of the five folds, performances obtained using the UNet-SW32 method are summarized in Table~\ref{tab:results-cross-fold-variability}. The recall and Dice score values achieved over the first fold are above the average performances by 3\%. Performances across each fold are relatively stable and contained within 4\% of the average values. The highest magnitude in difference exists for the patient-wise recall with 12\% between folds 3 and 4. However, it is worth noting that fold 4 exhibits the smallest amount of single or collocated lymph nodes, around 25\% fewer than in any other fold. Comparing the different slab-wise training schemes based on the first fold only appears trustworthy enough to identify the best performing approach in a reasonable amount of time.

\begin{table}[!ht]
\centering
\caption{Segmentation and instance detection performances collected for the five folds with the UNet-SW32 approach. The \# LN corresponds to the number of lymph nodes or clusters in each fold.}
\adjustbox{max width=\textwidth}{
\begin{tabular}{c|c|c|c|c|c|c|c|c|c}
Fold & PT & \# LN & Dice & Recall & Recall-PW & FPPP & GT-Perc \tabularnewline
\hline
1 & 0.3 & 459 & $67.34\pm14.49$ & 56.43 & $58.16\pm16.49$ & $4.04\pm1.85$ & $65.58\pm9.45$ \tabularnewline
2 & 0.3 & 410 & $64.47\pm11.25$ & 55.12 & $57.63\pm16.94$ & $5.83\pm3.34$ & $71.40\pm10.7$ \tabularnewline
3 & 0.3 & 451 & $61.88\pm14.83$ & 47.01 & $48.85\pm17.73$ & $5.83\pm2.94$ & $66.56\pm12.84$ \tabularnewline
4 & 0.3 & 330 & $63.21\pm16.40$ & 55.15 & $60.71\pm18.93$ & $9.37\pm4.56$ & $70.80\pm12.68$ \tabularnewline
5 & 0.3 & 472 & $64.58\pm16.27$ & 49.36 & $50.93\pm18.21$ & $4.08\pm2.26$ & $63.99\pm14.45$ \tabularnewline
\hline
Total & 0.3 & 2122 & $64.27\pm14.63$ & 52.40 & $55.23\pm17.98$ & $5.85\pm3.65$ & $67.68\pm12.32$ \tabularnewline
\end{tabular}
}
\label{tab:results-cross-fold-variability}
\end{table}


\begin{table}[b]
\caption{Overall performances obtained with the different slab-wise training configurations, for the first fold only. The fourth experiment, indicated by *, used a lower input resolution of $192\times128$\,voxels.}
\adjustbox{max width=\textwidth}{
\begin{tabular}{l|c|c|c|c|c|c|c|c|c|c}
Experiment & PT & Dice & Dice-TP & GT-Perc & Recall & Recall-PW & FPPP \tabularnewline
\hline
(i) UNet-SW32 & 0.3 & \boldmath{$67.34\pm14.49$} & \boldmath{$53.49\pm14.21$} & $65.58\pm09.45$ & \boldmath{$56.43$} & \boldmath{$58.16\pm16.49$} & $4.04\pm1.85$ \tabularnewline
(ii) UNet-SW64 & 0.4  & $65.86\pm14.18$ & $52.46\pm11.01$ & $62.77\pm08.23$ & $52.72$ & $56.94\pm15.66$ & $4.57\pm2.04$ \tabularnewline
(iii) AGUNet-SW64 & 0.2 & $64.24\pm13.55$ & $53.37\pm10.83$ & $77.12\pm09.51$ & $53.38$ & $55.49\pm15.34$ & \boldmath{$3.00\pm1.41$} \tabularnewline
(iv) AGUNet-SW64* & 0.2 & $62.86\pm13.24$ & $50.31\pm13.00$ & \boldmath{$78.55\pm09.72$} & $51.42$ & $53.96\pm16.82$ & $
3.74\pm2.16$ \tabularnewline
\end{tabular}
}
\label{tab:results-slabwise}
\end{table}

Performances over the first fold for the different models trained using a slab-wise strategy are reported in Table~\ref{tab:results-slabwise}. The best performing method regarding the Dice score and recall is the slab-wise U-Net architecture with 32 slices (experiment (i)), reaching up to 58\% patient-wise recall for all lymph nodes. Doubling the number of slices in the slab worsened all scores by about 2\%, probably indicating the need for more training samples (cf. experiments (i) and (ii)). Using more advanced architectures such as the attention-gated U-Net are showing improved precision with one less false positive per patient and improved GT-Perc, demonstrating improved pixel-wise segmentation around detected lymph nodes (cf. experiments (ii) and (iii)). However, this increase is accompanied by deteriorated recall and patient-wise recall values. By downsampling the input resolution, performances are diminished by 2\% at most as can be seen between experiments (iii) and (iv). Nonetheless, the minimal drop in performance for a faster training appears as a good trade-off to investigate training schemes or architecture designs in a more reasonable amount of time.

\subsection{Overall performance comparison}
For the overall performance comparison study, results have been averaged over the five folds and are reported in Table~\ref{tab:results-seg-overall}.
The best slab-wise approach is competitive in terms of recall when compared to full volume approaches, as can be seen between experiments (i) and (ii). In addition, and given the higher spatial resolution, the slab-wise approach is able to top pixel-wise segmentation performances with up to 64\% overall Dice score compared to 60\% with the best full volume approach (cf. experiments (i) and (iv)). Using more advanced architectures, in combination with the use of anatomical priors guiding, lead to a 3\% increase in recall and patient-wise recall together with a reduction of almost 1 FPPP (cf. experiments (ii) and (iii)). When ensembling a slab-wise and a full volume approach, recall and patient-wise recall performances are similar (cf. experiments (iii) and (vi)). However, only through ensembling can all the metrics reach close to their maximum simultaneously, and as such experiment (vi) is deemed to be our best performing method. The ensemble of a slab-wise U-Net using 32 slices and a full volume attention-guided U-Net using anatomical priors reaches a patient-wise recall of $58.8\%$, a Dice score of $64\%$, an extent of segmented ground truth up to $71\%$, for 5 false positives per patient. A 3D illustration for two patients, showing the ground truth and detected lymph nodes side-by-side, is provided in Fig.~\ref{fig:proba-results-illu-3D}, and where the four organs used for the anatomical priors guiding are also featured.


\begin{table}[!ht]
\caption{Overall performance comparison obtained for the different experiments carried out, averaged across the five folds. The abbreviations are: regular U-Net (UNet), attention-gated U-Net (AGUNet), dual attention guided U-Net (DAGUNet), anatomical priors guiding (APG), full volume (FV), and slab-wise with 32 slices (SW32).}
\adjustbox{max width=\textwidth}{
\begin{tabular}{l|c|c|c|c|c|c|c|c|c|c|}
Experiment & PT & Dice & Dice-TP & GT-Perc & Recall & Recall-PW & FPPP \tabularnewline
\hline
(i) UNet-SW32 & 0.3 & $64.27\pm14.63$ & \boldmath{$53.93\pm13.79$} & $67.68\pm12.32$ & $52.40$ & $55.23\pm17.98$ & $5.85\pm3.65$ \tabularnewline
(ii) AGUNet-FV & 0.2 & $59.48\pm13.48$ & $45.21\pm16.93$ & $72.73\pm12.17$ & $52.21$ & $55.86\pm19.07$ & $4.48\pm2.98$ \tabularnewline
(iii) AGUNet-FV-APG & 0.2 & $57.64\pm13.53$ & $42.13\pm15.88$ & \boldmath{$75.07\pm12.35$} & \boldmath{$55.42$} & $58.04\pm19.95$ & $4.48\pm2.95$ \tabularnewline
(iv) DAGUNet-FV-APG & 0.4 & $60.01\pm15.78$ & $45.89\pm16.51$ & $65.81\pm14.27$ & $50.80$ & $54.21\pm19.17$ & $4.19\pm2.87$ \tabularnewline
(v) Ensemble (i) \& (ii) & 0.4 & \boldmath{$65.01\pm12.75$} & $53.59\pm14.97$ & $69.70\pm11.44$ & $52.59$ & $56.02\pm18.15$ & $5.12\pm2.96$ \tabularnewline
(vi) Ensemble (i) \& (iii) & 0.4 & $64.00\pm13.21$ & $51.89\pm15.32$ & $71.01\pm11.46$ & $55.23$ & \boldmath{$58.79\pm19.72$} & $5.23\pm2.80$ \tabularnewline
(vii) Ensemble (i) \& (iv) & 0.6 & $64.16\pm14.78$ & $50.98\pm14.75$ & $62.02\pm12.74$ & $50.47$ & $53.66\pm19.13$ & \boldmath{$4.05\pm2.27$} \tabularnewline
\end{tabular}
}
\label{tab:results-seg-overall}
\end{table}

\begin{figure}[b]
\centering
\includegraphics[scale=2.55]{image6.png}
\caption{Representation in 3D of the results for two patients where the ground truth is shown to the left and the model output to the right. The four anatomical structures used for the anatomical priors guiding are also represented: esophagus (blue), azygos vein (red), subclavian arteries (yellow), and brachiocephalic veins (green).}
\label{fig:proba-results-illu-3D}
\end{figure}

For five different patients, one being featured per row and sampled from each fold, visual comparisons are provided in Fig.~\ref{fig:proba-results-illu} between the four main designs. The probability maps are set to the range $[0, 1]$ where cool colors indicate low confidence in the lymph node class and warm colors indicate high confidence, and the CT images are shown in the range $[-150, 500]$\,HU. In the first and fourth row, the segmentation performance is similar across all methods for the displayed lymph nodes. The effect of anatomical prior guiding can be witnessed in the third row, over the azygos vein to the left of the yellow lymph node. In the first two methods without guiding the upper part of the vein is by mistake partially segmented as being a lymph node. On the opposite, the azygos vein is left totally unsegmented when using anatomical prior guiding as can be shown with the last two methods. In the last row, the contrast is slightly different than from other patients, resulting in lymph nodes more challenging to detect. While the top lymph node (colored in yellow) is similarly segmented across all methods, two other lymph nodes are featured in the ground truth (colored in blue and magenta) with varying extent of segmentation. Given the difficulty to visually identify those lymph nodes, judging whether the manual ground truth has been under-segmented or the last two methods are over-segmenting, is extremely challenging.

\begin{figure}[!t]
\centering
\includegraphics[scale=0.65]{image7.png}
\caption{Prediction examples for a patient from each fold, one per line, for the four main designs: UNet-SW32, AGUNet, AGUNet-FV-APG, and DAGUNet-FV-APG.
Cool colors indicate low confidence in the lymph node class and warm colors indicate high confidence.}
\label{fig:proba-results-illu}
\end{figure}

\begin{table}[!b]
\centering
\caption{Quantification of false positive segmentation over the four anatomical structures used for priors guiding. For each of the four main designs, Dice scores averaged across the $120$ patients are reported.}
\adjustbox{max width=\textwidth}{
\begin{tabular}{l|c|c|c|c|}
Experiment & Esophagus & Azygos vein & Subclavian arteries & Brachiocephalic veins \tabularnewline
\hline
(i) UNet-SW32 & $3.54\%$ & $3.54\%$ & $0.49\%$ & $1.27\%$ \tabularnewline
(ii) AGUNet-FV & $6.87\%$ & $7.26\%$ & $1.20\%$ & $2.57\%$ \tabularnewline
(iii) AGUNet-FV-APG & $2.03\%$ & $1.89\%$ & $0.90\%$ & $0.60\%$ \tabularnewline
(iv) DAGUNet-FV-APG & $1.61\%$ & $1.64\%$ & $0.85\%$ & $0.48\%$ \tabularnewline
\end{tabular}
}
\label{tab:apg-benefit}
\end{table}

To quantify and fully appreciate the effect of anatomical priors guiding on the pixel-wise segmentation performance rather than just visually, a supplementary analysis has been completed. The Dice scores between a model prediction and each of the four organs making the anatomical priors are reported in Table~\ref{tab:apg-benefit}. Reported scores are predominantly lower for the two experiments making use of the priors when compared with the other two experiments. Without the use of priors, the slab-wise approach seems to fair better, with twice as low scores, than the full volume approach (cf. experiments (i) and (ii)). The higher spatial resolution from the slab-wise approach being surely profitable to distinguish between similar-looking voxels belonging to different anatomical structures. The benefit is even more apparent for over the esophagus and azygos vein, with a drastic reduction in false positive segmentation (cf. experiments (ii) and (iii)). 

Regarding the total processing time required to process a new CT volume with a single model, an average of $3.5\pm0$\,minutes has been calculated, using GPU support. For ensembling, models were run sequentially without any specific optimization, leading to a total processing time of $7.3\pm0.2$\,minutes. Interestingly, a similar performance is reached using only the CPU. Most of the processing time is dedicated to the different resampling and resizing operations, given the large resolution of the input CT volume, and performed by default on the CPU. Regarding pure inference, including model loading, only $8.4\pm0.4$\,seconds are necessary on average.

\subsection{Performance analysis over lymph nodes' characteristics}

\iffalse
\begin{table}[!ht]
\caption{Segmentation performances obtained with our best performing method for the three lymph node categories based on short-axis value. The first four measurements (left-most columns) are reported for all lymph nodes while the last four measurements (right-most columns) are reported for the lymph nodes belonging to relevant IASLC stations.}
\adjustbox{max width=\textwidth}{
\begin{tabular}{c||c|c|c|c||c|c|c|c|}
 \multirow{2}{*}{Short-axis} & \multicolumn{4}{c||}{All stations} & \multicolumn{4}{c|}{Relevant stations} \tabularnewline
 \cline{2-9}
 & Dice-TP & GT-Perc & Recall & Recall-PW & Dice-TP & GT-Perc & Recall & Recall-PW \tabularnewline
\hline
\hline
All & $51.89\pm15.32$ & $71.01\pm11.46$ & 55.23 & $58.79\pm19.72$ & $51.68\pm15.54$ & $71.59\pm11.77$ & $61.50$ & $65.46\pm20.14$ \tabularnewline
$\geq7\,mm$ & $55.03\pm15.31$ & $74.20\pm12.05$ & 75.63 & $79.59\pm18.41$ & $54.80\pm15.52$ & $74.69\pm12.46$ & $80.53$ & $83.33\pm18.05$ \tabularnewline
$\geq10\,mm$ & $59.02\pm16.48$ & $80.04\pm12.60$ & 85.73 & $88.96\pm14.75$ & $58.54\pm16.99$ & $80.55\pm12.54$ & $90.73$ & $92.10\pm14.43$ \tabularnewline
\end{tabular}
}
\label{tab:results-size-stations}
\end{table}
\fi

\begin{table}[!ht]
\caption{Segmentation performances obtained with our best performing method for the three lymph node categories based on short-axis value. The first four measurements (left-most columns) are reported for all lymph nodes while the last four measurements (right-most columns) are reported for the lymph nodes belonging to relevant IASLC stations.}
\adjustbox{max width=\textwidth}{
\begin{tabular}{c|cccc|cccc|}
 \multirow{2}{*}{Short-axis} & \multicolumn{4}{c|}{All stations} & \multicolumn{4}{c|}{Relevant stations} \tabularnewline
 \cline{2-9}
 & Dice-TP & GT-Perc & Recall & Recall-PW & Dice-TP & GT-Perc & Recall & Recall-PW \tabularnewline
\hline
All & $51.89\pm15.32$ & $71.01\pm11.46$ & 55.23 & $58.79\pm19.72$ & $51.68\pm15.54$ & $71.59\pm11.77$ & $61.50$ & $65.46\pm20.14$ \tabularnewline
$\geq7\,mm$ & $55.03\pm15.31$ & $74.20\pm12.05$ & 75.63 & $79.59\pm18.41$ & $54.80\pm15.52$ & $74.69\pm12.46$ & $80.53$ & $83.33\pm18.05$ \tabularnewline
$\geq10\,mm$ & $59.02\pm16.48$ & $80.04\pm12.60$ & 85.73 & $88.96\pm14.75$ & $58.54\pm16.99$ & $80.55\pm12.54$ & $90.73$ & $92.10\pm14.43$ \tabularnewline
\end{tabular}
}
\label{tab:results-size-stations}
\end{table}

\begin{figure}[!b]
\centering
\includegraphics[scale=0.51]{image8.png}~\includegraphics[scale=0.51]{image9.png}
\caption{Lymph nodes recall performance per primary station for our best approach. To the left all lymph nodes are considered, to the right only the lymph nodes with a short-axis diameter $\geq10$\,mm are considered. The red color represents the total amount of lymph nodes and the green color the amount positively detected.}
\label{fig:recall-per-station}
\end{figure}

From the best performing approach, ensemble of UNet-SW32 and AGUNet-FV-APG, performances based on lymph nodes' primary station and short-axis diameter are reported in Table.~\ref{tab:results-size-stations}.
When considering lymph nodes of all sizes, a $6$\% recall increase can be witnessed when discarding the lymph nodes located in regions not corresponding to any IASLC station or in station 1 (i.e., relevant stations category). Those lymph nodes are either heavily under-represented in our dataset, or simply not included during sample preprocessing, explaining the model's worst performance. A significant recall improvement of $20$\% can be acknowledged when focusing on lymph nodes with a short-axis diameter $\geq7$\,mm. Regarding the most clinically relevant lymph nodes, with a short-axis diameter $\geq10$\,mm and potentially malignant following the RECIST criterion, an overall recall of $90.73$\% is reached along with a patient-wise recall of $92$\%.
The pixel-wise segmentation performances are comparatively worse, with an average Dice-TP of $59$\% at best. However, the score is not directly representative of the segmentation quality and is heavily impacted by the process including the connected components approach and pairing strategy. A Dice score computed between a single lymph node and a cluster of collocated lymph nodes will be inherently low, even if the pixel-segmentation is accurate over the different lymph nodes in the cluster. If anything, this value is an indication of the method's struggles to perform instance disambiguation. Conversely, the GT-Perc score is a better metric to assess the total extent of properly segmented lymph node voxels. On average, up to $80$\% of the voxels belonging to the clinically relevant lymph nodes have been successfully segmented. While the noisiness in the results is not factored in here, the combination of the different metrics (i.e., Dice-TP, GT-Perc, and FPPP) should be enough to appreciate the quality of the pixel-wise segmentation.


Leaving laterality aside, the recall performances are reported with respect to each of the IASLC station in Fig.~\ref{fig:recall-per-station}.
Considering only the clinically relevant lymph nodes (i.e., right sub-plot), no major recall discrepancy can be identified across the different stations. Even though the distribution of lymph nodes featured in each station is heavily unbalanced, especially between station 4 and 9, the model's ability to segment the largest lymph nodes has not been hindered.
When including lymph nodes of all short-axis diameters (i.e., left sub-plot), stations 2, 3, and 8 are standing out with a recall below $60$\%. Station 2 being the most populated station, the lower recall percentage cannot be explained by an under-representation or data imbalance rationalization. Having the possibility to train our models with a higher input resolution should help segmenting the smallest and challenging lymph nodes.


An investigation over the pixel-wise segmentation quality across the different stations is provided in Fig.~\ref{fig:detected-dice-per-station}. The average GT-Perc scores (right sub-plot) are stable across the different stations with values around or above $80$\%. Only station 7 seems to be standing out with a more compact range of scores, far from the few outliers. A similar general behaviour can be observed for the Dice-TP scores (left sub-plot).


\begin{figure}[!t]
\centering
\includegraphics[scale=0.57]{image10.png}~\includegraphics[scale=0.57]{image11.png}
\caption{Segmentation performance for lymph nodes with a short-axis diameter $\geq10$\,mm and featured in the relevant stations category, obtained with our best performing approach. The Dice-TP scores are reported to the left and the GT-Perc scores to the right.}
\label{fig:detected-dice-per-station}
\end{figure}

\subsection{Ground truth quality assessment over a benchmark subset}
For assessing the quality of the ground truth, a direct comparison between assigned stations for the $15$ patients of the benchmark subset has been performed. Out of the $363$ lymph nodes featured,  $312$ instances were attributed a perfect grade as both the expert and medical trainee were in agreement, amounting to about $86$\% of all cases. A good grade was associated with $33$ instances (around $9$\%), as the expert believed the primary and secondary stations assigned by the trainee were to be swapped. Given the loose definition from the IASLC guidelines, capturing exactly the quantity of a lymph node lying in each station is impossible solely from a CT volume. As such, $95$\% of all lymph nodes were considered to be correctly assigned with their main stations (up to three). Lastly, $18$ instances were assigned a wrong station according to the expert, representing slightly less than $5$\% of all cases.
The vast majority of confused cases revolved around a mistaken assignment to station 10 whereby lymph nodes should have been assigned to station 4 (in 7 cases), station 5 (in 4 cases), or station 6 and 12 (in 1 case each).


Regarding the quality of the pixel-wise segmentation, the task was not asked to be performed by the second expert, as being too time-consuming. From a thorough eye-balling, no clinically relevant lymph node was mentioned as having been overlooked. On three occurrences, the segmentation was suggested to be refined as more than one lymph node could be identified from the segmented cluster.


For completeness, the performances obtained on the benchmark subset, with our best performing method, are reported in Tab.~\ref{tab:results-size-stations_benchmark}. Not indicated in the table, an FPPP rate of $4.73\pm2.52$ was obtained on average over the $15$ patients.

\iffalse
\begin{table}[!t]
\caption{Segmentation performances obtained with our best performing method for the three lymph node categories based on short-axis value. The first four measurements (left-most columns) are reported for all lymph nodes while the last four measurements (right-most columns) are reported for the lymph nodes belonging to relevant IASLC stations. Only for the 15 patients of the benchmark subset.}
\adjustbox{max width=\textwidth}{
\begin{tabular}{c||c|c|c|c||c|c|c|c|}
\multirow{2}{*}{Short-axis} & \multicolumn{4}{c||}{All stations} & \multicolumn{4}{c|}{Relevant stations} \tabularnewline
\cline{2-9}
 & Dice-TP & GT-Perc & Recall & Recall-PW & Dice-TP & GT-Perc & Recall & Recall-PW \tabularnewline
\hline
All & $44.75\pm13.52$ & $53.13\pm19.44$ & $46.42$ & $46.47\pm11.98$ & $44.49\pm13.07$ & $52.61\pm19.24$ & $56.02$ & $58.39\pm16.19$ \tabularnewline
$\geq7\,mm$ & $47.92\pm12.89$ & $55.78\pm20.74$ & $72.67$ & $80.01\pm19.58$ & $47.66\pm12.50$ & $55.26\pm20.63$ & $78.26$ & $81.78\pm17.04$ \tabularnewline
$\geq10\,mm$ & $53.42\pm16.70$ & $58.86\pm22.78$ & $82.72$ & $88.60\pm15.57$ & $53.14\pm16.47$ & $58.30\pm22.75$ & $89.19$ & $90.49\pm13.11$ \tabularnewline
\end{tabular}
}
\label{tab:results-size-stations_benchmark}
\end{table}
\fi

\begin{table}[!t]
\caption{Segmentation performances obtained with our best performing method for the three lymph node categories based on short-axis value. The first four measurements (left-most columns) are reported for all lymph nodes while the last four measurements (right-most columns) are reported for the lymph nodes belonging to relevant IASLC stations. Only for the 15 patients of the benchmark subset.}
\adjustbox{max width=\textwidth}{
\begin{tabular}{c|cccc|cccc|}
\multirow{2}{*}{Short-axis} & \multicolumn{4}{c|}{All stations} & \multicolumn{4}{c|}{Relevant stations} \tabularnewline
\cline{2-9}
 & Dice-TP & GT-Perc & Recall & Recall-PW & Dice-TP & GT-Perc & Recall & Recall-PW \tabularnewline
\hline
All & $44.75\pm13.52$ & $53.13\pm19.44$ & $46.42$ & $46.47\pm11.98$ & $44.49\pm13.07$ & $52.61\pm19.24$ & $56.02$ & $58.39\pm16.19$ \tabularnewline
$\geq7\,mm$ & $47.92\pm12.89$ & $55.78\pm20.74$ & $72.67$ & $80.01\pm19.58$ & $47.66\pm12.50$ & $55.26\pm20.63$ & $78.26$ & $81.78\pm17.04$ \tabularnewline
$\geq10\,mm$ & $53.42\pm16.70$ & $58.86\pm22.78$ & $82.72$ & $88.60\pm15.57$ & $53.14\pm16.47$ & $58.30\pm22.75$ & $89.19$ & $90.49\pm13.11$ \tabularnewline
\end{tabular}
}
\label{tab:results-size-stations_benchmark}
\end{table}

\section{Discussion}
\label{sec:discussion}
Solely from the visual inspection of a CT volume, a perfect identification, pixel-wise segmentation, and station mapping of all lymph nodes is arguably close to impossible for various reasons. The resolution in CT acquisitions is relatively correct, but inferior to what ultrasound can achieve during the EBUS procedure. The timing and quality of the contrast uptake is volatile, resulting in some regions visually exhibiting the same characteristics and attenuation as lymph nodes (e.g., mucus, fluid, or other soft tissues), often without clear boundaries. Only a thorough biopsy sample from every lymph node candidate identified on CT would ascertain the perfection of the ground truth, which is unrealistic to achieve in practice. Therefore, a conservative annotation approach was adopted whereby every suspicious region has been labelled as a lymph node, potentially engendering a bias towards hyper-detection. Similarly, visible lymph nodes of any short-axis diameter were annotated if possible, even smaller than the recommended RECIST criterion of $10$\,mm. The inclusion of smaller lymph nodes can be seen as a very efficient data augmentation approach especially for location, either bypassing or complementing heavier data augmentation transforms during training (e.g., zoom, affine, or perspective operations). In addition, clinicians tend to find all PET-positive lymph nodes to be relevant, regardless of size, and often the number and pattern of enlargement in even smaller lymph nodes is looked at.\\
Regarding station mapping, the guidelines prescribed by the IASLC are approximate since construed relatively to the surrounding anatomical structures in the mediastinum, leaving room for interpretation and speculating. In addition to constant evolution of the guidelines through yearly updates, lymph nodes, either expressed as singular entities or within a cluster, can be featured in multiple stations simultaneously.
For all those reasons, we do not claim exact segmentation and station mapping in our ground truth. However, from the results of the fourth validation study, the deviation between the medical trainee and expert was minimal. Some trust regarding the manual annotations can then be granted for the other patients not proofed by the expert. In any case, for this nontrivial task a perfect ground-truth cannot be expected, and we believe this annotation work to be a step in the right direction, sufficient to get insights over the performance in each station. Stemming from this work, multi-task architectures simultaneously performing pixel-wise segmentation, instance detection, and station classification could be investigated.


Regarding the architecture designs or training schemes investigated, a mild impact could be appreciated from the simpler or more complex operations, with a 5-10\% variation across the different metrics studied. The main limitation to get better performances is most likely coming from the dataset itself rather than the chosen methods. While a hefty number of lymph nodes are featured, a total of $120$ patients might not represent enough diversity, especially when training full volume approaches whereby one patient equates to only one training sample. Lymph nodes do exhibit a wide range of expressions (i.e., shape, size, location), a clear imbalance in the stations distributions has been highlighted, and a disparity in contrast-enhancing has also been witnessed. Therefore, access to a dataset orders of magnitude larger would be needed but it represents a tremendous data collection and annotation workload.
Performing ad-hoc ensembling as post-processing enables to benefit from the higher spatial resolution generated by the slab-wise models and global relationships from full volume models. As it is, the ensemble approach is favoring recall and GT-Perc in essence but at the expanse of precision. Smarter and end-to-end consensus designs should be investigated in-depth, using more than simply two models, which would require more data and induce a longer training time.


Using anatomical priors during the training process partly had the expected behaviour, whereby less false positive segmentation was predicted over the esophagus and azygos vein. At the same time, side-effects were also witnessed from lower predicted probabilities over lymph nodes close to those anatomical structures and with similar attenuation values. In order to purely favor instance detection recall, anatomical priors might be leveraged as well in post-processing. The larger number of voxels predicted with a high probability to belong to the lymph node class could be refined by applying a mask containing the location of every other know anatomical structure in the mediastinum. The number of false positives per patient would then be drastically reduced, the pixel-wise segmentation over the lymph nodes refined, and the overall patient-wise recall kept high. With such strategies, well-performing models are required for at least fifteen to twenty anatomical structures. Considering a standalone model for each anatomical structure, the total processing time for a new CT patient would be forcibly longer yet not detrimental as real-time processing is not a requirement for this modality. This post-processing step could either be performed as a simple masking, or end-to-end through a shallow refinement network.
Be it as it may, we believe there is strong potential in further investigating anatomical priors guiding, which would circumvent the need for any refinement or post-processing step. More training samples, and especially the knowledge of more than four other anatomical structures, appear mandatory to proceed.


For the performance assessment, with results generated by a pixel-wise segmentation architecture deprived of any instance detection component, a direct comparison to the raw ground truth was deemed unfair. In many cases involving collocated lymph nodes, no apparent boundary is clearly visible between the different elements of a cluster. The use of a connected component approach over a thresholded prediction map, and the conceivable existence of noise in the pixel-wise segmentation, prevents from a fair one-to-one mapping between prediction and ground truth.
Besides the patient-wise Dice score to assess the quality of the segmentation, reporting instance-wise segmentation performances in two different ways gives further insights. With Dice-TP, the segmentation quality and correctness in instance pairing can be simultaneously assessed. The full extent of segmented ground truth, disregarding instantiating consideration and false positives, can be gauged by GT-Perc scores. Given the challenges in reporting instance detection performances from results of a pure segmentation method, all the metrics considered in the validation studies should be sufficient to cover all aspects and properly enable to put our results in perspective.\\
Given the extremely long time required to perform a full 5-fold cross-validation with slab-wise approaches, nearing 60 days, only a few experiments could reasonably be carried out. Given the low variability in performance across the different folds, comparing architectures and designs over a single fold is arguably enough to identify the best performing ones. Interestingly, models trained using full volumes as input are performing nearly as well as slab-wise approaches, with the benefit to train at a much faster pace. Focusing on improving full volume approaches might be more relevant in the future as experiments can be carried out with less time required.


When considering clinically relevant lymph nodes (i.e., with a short-axis diameter $\geq10$\,mm), and featured in relevant IASLC stations, a patient-wise recall of $92$\%, a segmentation extent of $81$\% on average, and a FPPP ratio of $5$, was reached. Even though the distribution of lymph nodes per station is heavily uneven, the segmentation and detection performances are smooth and stable across all stations. Considering a potential use in a clinical setting, such results are encouraging as a first step to generate report and compute automatic measurements. The main limitation is the inability to properly detect instances and separate collocated lymph nodes. In its current status, our model can be used as preprocessing to bring attention to mediastinal areas potentially featuring lymph nodes, letting the clinical team decide on the mandatory zones where to perform biopsy.


In this work, the knowledge regarding the stations was merely used as a tool to drive the interpretation of the results, with the intent to fine-tune the training schemes in case of need. From the large room given to interpretation, and the proportion of lymph nodes to lie in at least two stations simultaneously, training an end-to-end multi-task architecture for segmentation and station classification appears to this day too challenging. Collecting more data ranks high in the list of future tasks, enabling the possibility to investigate smarter balancing or training sampling schemes. As intermediate solution, station classification could be explored as a refinement step from the results of the pixel-wise segmentation, either in an ad-hoc fashion or by using another shallower architecture. Finally, investigating multi-task architecture in 3D (e.g., Mask R-CNN, YOLACT) is a possibility but would require more powerful GPUs, and a larger dataset.


\section{Conclusion}
\label{sec:conclusion}
The segmentation of mediastinal lymph nodes in CT volumes has been investigated in this study with a focus on 3D neural network architectures and ensembles. Simple and more complex architectures using U-Net as backbone were explored, leveraging either the entire CT volume after heavy downsampling or slab-wise. To benefit from the advantages of each strategy, post-training ensembles enable to generate the best segmentation and instance detection performances. Similarly, anatomical priors guiding reduces false positive predictions over other anatomical structures with similar attenuation values. For clinically relevant lymph nodes, a patient-wise recall of $92$\% was reached for a ratio of up to 5 false positives per patient, with steady performances across the different IASLC stations. In future work, being able to dissociate collocated lymph nodes and properly perform instance detection using a multi-task architecture appears of interest. In addition, better leveraging the station information is of importance to transform the method into a proper and trustworthy diagnostic tool. Finally, increasing the dataset size is needed to gain more data diversity which would in turn improve overall performances.

\subsection*{Disclosures}
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\\
Informed consent was obtained from all individual participants included in the study.

\subsection* {Acknowledgments}
This work was funded by the Norwegian National Advisory Unit for Ultrasound and Image-Guided Therapy (usigt.org) at St. Olavs hospital, Trondheim, Norway.


\bibliographystyle{unsrt}  
\bibliography{references}


\end{document}

article.bbl
\begin{thebibliography}{10}

\bibitem{falk2010}
Williams~C Falk~S.
\newblock Chapter 1.
\newblock {\em Lung Cancer: the facts (3rd ed.)}, pages pp3--4, 2010.

\bibitem{eisenhauer2009new}
Elizabeth~A Eisenhauer, Patrick Therasse, Jan Bogaerts, Lawrence~H Schwartz,
  D~Sargent, Robert Ford, Janet Dancey, S~Arbuck, Steve Gwyther, Margaret
  Mooney, et~al.
\newblock New response evaluation criteria in solid tumours: revised recist
  guideline (version 1.1).
\newblock {\em European journal of cancer}, 45(2):228--247, 2009.

\bibitem{schwartz2009evaluation}
LH~Schwartz, J~Bogaerts, R~Ford, L~Shankar, P~Therasse, S~Gwyther, and
  EA~Eisenhauer.
\newblock Evaluation of lymph nodes with recist 1.1.
\newblock {\em European journal of cancer}, 45(2):261--267, 2009.

\bibitem{el2014international}
Ahmed~H El-Sherief, Charles~T Lau, Carol~C Wu, Richard~L Drake, Gerald~F
  Abbott, and Thomas~W Rice.
\newblock International association for the study of lung cancer (iaslc) lymph
  node map: radiologic review with ct illustration.
\newblock {\em Radiographics}, 34(6):1680--1691, 2014.

\bibitem{sorger2017multimodal}
Hanne Sorger, Erlend~Fagertun Hofstad, Tore Amundsen, Thomas Lang{\o}, Janne
  Beate~Lervik Bakeng, and H{\aa}kon~Olav Leira.
\newblock A multimodal image guiding system for navigated ultrasound
  bronchoscopy (ebus): A human feasibility study.
\newblock {\em PloS one}, 12(2):e0171841, 2017.

\bibitem{mcerlean2013intra}
Aoife McErlean, David~M Panicek, Emily~C Zabor, Chaya~S Moskowitz, Richard
  Bitar, Robert~J Motzer, Hedvig Hricak, and Michelle~S Ginsberg.
\newblock Intra-and interobserver variability in ct measurements in oncology.
\newblock {\em Radiology}, 269(2):451--459, 2013.

\bibitem{long2015fully}
Jonathan Long, Evan Shelhamer, and Trevor Darrell.
\newblock Fully convolutional networks for semantic segmentation.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 3431--3440, 2015.

\bibitem{ronneberger2015u}
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
\newblock U-net: Convolutional networks for biomedical image segmentation.
\newblock In {\em International Conference on Medical image computing and
  computer-assisted intervention}, pages 234--241. Springer, 2015.

\bibitem{zhou2017fixed}
Yuyin Zhou, Lingxi Xie, Wei Shen, Yan Wang, Elliot~K Fishman, and Alan~L
  Yuille.
\newblock A fixed-point model for pancreas segmentation in abdominal ct scans.
\newblock In {\em International conference on medical image computing and
  computer-assisted intervention}, pages 693--701. Springer, 2017.

\bibitem{kavur2020chaos}
A~Emre Kavur, N~Sinem Gezer, Mustafa Bar{\i}{\c{s}}, Pierre-Henri Conze,
  Vladimir Groza, Duc~Duy Pham, Soumick Chatterjee, Philipp Ernst, Sava{\c{s}}
  {\"O}zkan, Bora Baydar, et~al.
\newblock Chaos challenge--combined (ct-mr) healthy abdominal organ
  segmentation.
\newblock {\em arXiv preprint arXiv:2001.06535}, 2020.

\bibitem{simpson2019large}
Amber~L. Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan
  Farahani, Bram van Ginneken, Annette Kopp-Schneider, Bennett~A. Landman,
  Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald~M. Summers, Patrick
  Bilic, Patrick~F. Christ, Richard K.~G. Do, Marc Gollub, Jennifer
  Golia-Pernicka, Stephan~H. Heckers, William~R. Jarnagin, Maureen~K. McHugo,
  Sandy Napel, Eugene Vorontsov, Lena Maier-Hein, and M.~Jorge Cardoso.
\newblock A large annotated medical image dataset for the development and
  evaluation of segmentation algorithms, 2019.

\bibitem{vesal20192d}
Sulaiman Vesal, Nishant Ravikumar, and Andreas Maier.
\newblock A 2d dilated residual u-net for multi-organ segmentation in thoracic
  ct.
\newblock {\em arXiv preprint arXiv:1905.07710}, 2019.

\bibitem{wang2019abdominal}
Yan Wang, Yuyin Zhou, Wei Shen, Seyoun Park, Elliot~K Fishman, and Alan~L
  Yuille.
\newblock Abdominal multi-organ segmentation with organ-attention networks and
  statistical fusion.
\newblock {\em Medical image analysis}, 55:88--102, 2019.

\bibitem{zhuang2016multi}
Xiahai Zhuang and Juan Shen.
\newblock Multi-scale patch and multi-modality atlases for whole heart
  segmentation of mri.
\newblock {\em Medical image analysis}, 31:77--87, 2016.

\bibitem{setio2016pulmonary}
Arnaud Arindra~Adiyoso Setio, Francesco Ciompi, Geert Litjens, Paul Gerke,
  Colin Jacobs, Sarah~J Van~Riel, Mathilde Marie~Winkler Wille, Matiullah
  Naqibullah, Clara~I S{\'a}nchez, and Bram van Ginneken.
\newblock Pulmonary nodule detection in ct images: false positive reduction
  using multi-view convolutional networks.
\newblock {\em IEEE transactions on medical imaging}, 35(5):1160--1169, 2016.

\bibitem{cciccek20163d}
{\"O}zg{\"u}n {\c{C}}i{\c{c}}ek, Ahmed Abdulkadir, Soeren~S Lienkamp, Thomas
  Brox, and Olaf Ronneberger.
\newblock 3d u-net: learning dense volumetric segmentation from sparse
  annotation.
\newblock In {\em International conference on medical image computing and
  computer-assisted intervention}, pages 424--432. Springer, 2016.

\bibitem{kamnitsas2016deepmedic}
Konstantinos Kamnitsas, Enzo Ferrante, Sarah Parisot, Christian Ledig, Aditya~V
  Nori, Antonio Criminisi, Daniel Rueckert, and Ben Glocker.
\newblock Deepmedic for brain tumor segmentation.
\newblock In {\em International workshop on Brainlesion: Glioma, multiple
  sclerosis, stroke and traumatic brain injuries}, pages 138--149. Springer,
  2016.

\bibitem{he2017mask}
Kaiming He, Georgia Gkioxari, Piotr Doll{\'a}r, and Ross Girshick.
\newblock Mask r-cnn.
\newblock In {\em Proceedings of the IEEE international conference on computer
  vision}, pages 2961--2969, 2017.

\bibitem{redmon2018yolov3}
Joseph Redmon and Ali Farhadi.
\newblock Yolov3: An incremental improvement.
\newblock {\em arXiv preprint arXiv:1804.02767}, 2018.

\bibitem{oda2017hessian}
Hirohisa Oda, Kanwal~K Bhatia, Masahiro Oda, Takayuki Kitasaka, Shingo Iwano,
  Hirotoshi Homma, Hirotsugu Takabatake, Masaki Mori, Hiroshi Natori, Julia~A
  Schnabel, et~al.
\newblock Hessian-assisted supervoxel: structure-oriented voxel clustering and
  application to mediastinal lymph node detection from ct volumes.
\newblock In {\em Medical Imaging 2017: Computer-Aided Diagnosis}, volume
  10134, page 101341D. International Society for Optics and Photonics, 2017.

\bibitem{oda2017automated}
Hirohisa Oda, Kanwal~K Bhatia, Masahiro Oda, Takayuki Kitasaka, Shingo Iwano,
  Hirotoshi Homma, Hirotsugu Takabatake, Masaki Mori, Hiroshi Natori, Julia~A
  Schnabel, et~al.
\newblock Automated mediastinal lymph node detection from ct volumes based on
  intensity targeted radial structure tensor analysis.
\newblock {\em Journal of Medical Imaging}, 4(4):044502, 2017.

\bibitem{paing2019automatic}
May~Phu Paing, Chuchart Pintavirooj, Supan Tungjitkusolmun, Kyi~Pyar Win, and
  Kazuhiko Hamamoto.
\newblock Automatic detection of mediastinal lymph nodes using 3d convolutional
  neural network.
\newblock In {\em Proceedings of the 2019 4th International Conference on
  Biomedical Imaging, Signal Processing}, pages 26--31, 2019.

\bibitem{nogues2016automatic}
Isabella Nogues, Le~Lu, Xiaosong Wang, Holger Roth, Gedas Bertasius, Nathan
  Lay, Jianbo Shi, Yohannes Tsehay, and Ronald~M Summers.
\newblock Automatic lymph node cluster segmentation using holistically-nested
  neural networks and structured optimization in ct images.
\newblock In {\em International Conference on Medical Image Computing and
  Computer-Assisted Intervention}, pages 388--397. Springer, 2016.

\bibitem{roth2014new}
Holger~R Roth, Le~Lu, Ari Seff, Kevin~M Cherry, Joanne Hoffman, Shijun Wang,
  Jiamin Liu, Evrim Turkbey, and Ronald~M Summers.
\newblock A new 2.5 d representation for lymph node detection using random sets
  of deep convolutional neural network observations.
\newblock In {\em International conference on medical image computing and
  computer-assisted intervention}, pages 520--527. Springer, 2014.

\bibitem{liu2014mediastinal}
Jiamin Liu, Jocelyn Zhao, Joanne Hoffman, Jianhua Yao, Weidong Zhang, Evrim~B
  Turkbey, Shijun Wang, Christine Kim, and Ronald~M Summers.
\newblock Mediastinal lymph node detection on thoracic ct scans using spatial
  prior from multi-atlas label fusion.
\newblock In {\em Medical Imaging 2014: Computer-Aided Diagnosis}, volume 9035,
  page 90350M. International Society for Optics and Photonics, 2014.

\bibitem{liu2016mediastinal}
Jiamin Liu, Joanne Hoffman, Jocelyn Zhao, Jianhua Yao, Le~Lu, Lauren Kim,
  Evrim~B Turkbey, and Ronald~M Summers.
\newblock Mediastinal lymph node detection and station mapping on chest ct
  using spatial priors and random forest.
\newblock {\em Medical physics}, 43(7):4362--4374, 2016.

\bibitem{oda2018dense}
Hirohisa Oda, Holger~R Roth, Kanwal~K Bhatia, Masahiro Oda, Takayuki Kitasaka,
  Shingo Iwano, Hirotoshi Homma, Hirotsugu Takabatake, Masaki Mori, Hiroshi
  Natori, et~al.
\newblock Dense volumetric detection and segmentation of mediastinal lymph
  nodes in chest ct images.
\newblock In {\em Medical Imaging 2018: Computer-Aided Diagnosis}, volume
  10575, page 1057502. International Society for Optics and Photonics, 2018.

\bibitem{bouget2019semantic}
David Bouget, Arve J{\o}rgensen, Gabriel Kiss, Haakon~Olav Leira, and Thomas
  Lang{\o}.
\newblock Semantic segmentation and detection of mediastinal lymph nodes and
  anatomical structures in ct data for lung cancer staging.
\newblock {\em International journal of computer assisted radiology and
  surgery}, 14(6):977--986, 2019.

\bibitem{li2020deep}
Zhe Li and Yong Xia.
\newblock Deep reinforcement learning for weakly-supervised lymph node
  segmentation in ct images.
\newblock {\em IEEE Journal of Biomedical and Health Informatics}, 2020.

\bibitem{xu2020focal}
Guoping Xu, Hanqiang Cao, Youli Dong, Chunyi Yue, Kexin Li, and Yubing Tong.
\newblock Focal loss function based deeplabv3+ for pathological lymph node
  segmentation on pet/ct.
\newblock In {\em Proceedings of the 2020 2nd International Conference on
  Intelligent Medicine and Image Processing}, pages 24--28, 2020.

\bibitem{zhu2020lymph}
Zhuotun Zhu, Dakai Jin, Ke~Yan, Tsung-Ying Ho, Xianghua Ye, Dazhou Guo,
  Chun-Hung Chao, Jing Xiao, Alan Yuille, and Le~Lu.
\newblock Lymph node gross tumor volume detection and segmentation via
  distance-based gating using 3d ct/pet imaging in radiotherapy.
\newblock In {\em International Conference on Medical Image Computing and
  Computer-Assisted Intervention}, pages 753--762. Springer, 2020.

\bibitem{bolya2019yolact}
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong~Jae Lee.
\newblock Yolact: Real-time instance segmentation.
\newblock In {\em Proceedings of the IEEE/CVF International Conference on
  Computer Vision}, pages 9157--9166, 2019.

\bibitem{he2016deep}
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
\newblock Deep residual learning for image recognition.
\newblock In {\em Proceedings of the IEEE conference on computer vision and
  pattern recognition}, pages 770--778, 2016.

\bibitem{bai2017human}
Wenjia Bai, Matthew Sinclair, Giacomo Tarroni, Ozan Oktay, Martin Rajchl,
  Ghislain Vaillant, Aaron~M Lee, Nay Aung, Elena Lukaschuk, Mihir~M Sanghvi,
  et~al.
\newblock Human-level cmr image analysis with deep fully convolutional
  networks.
\newblock 2017.

\bibitem{liao2019evaluate}
Fangzhou Liao, Ming Liang, Zhe Li, Xiaolin Hu, and Sen Song.
\newblock Evaluate the malignancy of pulmonary nodules using the 3-d deep leaky
  noisy-or network.
\newblock {\em IEEE transactions on neural networks and learning systems},
  30(11):3484--3495, 2019.

\bibitem{sang2020pcanet}
Haiwei Sang, Qiuhao Zhou, and Yong Zhao.
\newblock Pcanet: Pyramid convolutional attention network for semantic
  segmentation.
\newblock {\em Image and Vision Computing}, 103:103997, 2020.

\bibitem{abraham2019novel}
Nabila Abraham and Naimul~Mefraz Khan.
\newblock A novel focal tversky loss function with improved attention u-net for
  lesion segmentation.
\newblock In {\em 2019 IEEE 16th International Symposium on Biomedical Imaging
  (ISBI 2019)}, pages 683--687. IEEE, 2019.

\bibitem{chen2017rethinking}
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam.
\newblock Rethinking atrous convolution for semantic image segmentation.
\newblock {\em arXiv preprint arXiv:1706.05587}, 2017.

\bibitem{sinha2020multi}
Ashish Sinha and Jose Dolz.
\newblock Multi-scale self-guided attention for medical image segmentation.
\newblock {\em IEEE Journal of Biomedical and Health Informatics}, 2020.

\bibitem{lee2015deeply}
Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu.
\newblock Deeply-supervised nets.
\newblock In {\em Artificial intelligence and statistics}, pages 562--570,
  2015.

\bibitem{jetley2018learn}
Saumya Jetley, Nicholas~A. Lord, Namhoon Lee, and Philip H.~S. Torr.
\newblock Learn to pay attention, 2018.

\bibitem{fu2019dual}
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing
  Lu.
\newblock Dual attention network for scene segmentation.
\newblock In {\em Proceedings of the IEEE Conference on Computer Vision and
  Pattern Recognition}, pages 3146--3154, 2019.

\bibitem{bouget2021meningioma}
David Bouget, Andr{\'e} Pedersen, Sayied Abdol~Mohieb Hosainey, Ole Solheim,
  and Ingerid Reinertsen.
\newblock Meningioma segmentation in t1-weighted mri leveraging global context
  and attention mechanisms.
\newblock {\em arXiv preprint arXiv:2101.07715}, 2021.

\bibitem{feng2020brain}
Xue Feng, Nicholas~J Tustison, Sohil~H Patel, and Craig~H Meyer.
\newblock Brain tumor segmentation using an ensemble of 3d u-nets and overall
  survival prediction using radiomic features.
\newblock {\em Frontiers in Computational Neuroscience}, 14:25, 2020.

\bibitem{hofmanninger2020automatic}
Johannes Hofmanninger, Florian Prayer, Jeanny Pan, Sebastian Rohrich, Helmut
  Prosch, and Georg Langs.
\newblock Automatic lung segmentation in routine imaging is a data diversity
  problem, not a methodology problem.
\newblock {\em arXiv preprint arXiv:2001.11767}, 2020.

\bibitem{reynisson2015airway}
Pall~Jens Reynisson, Marta Scali, Erik Smistad, Erlend~Fagertun Hofstad,
  H{\aa}kon~Olav Leira, Frank Lindseth, Toril~Anita Nagelhus~Hernes, Tore
  Amundsen, Hanne Sorger, and Thomas Lang{\o}.
\newblock Airway segmentation and centerline extraction from thoracic
  ct--comparison of a new method to state of the art commercialized methods.
\newblock {\em PloS one}, 10(12):e0144282, 2015.

\bibitem{imgaug}
Alexander~B. Jung, Kentaro Wada, Jon Crall, Satoshi Tanaka, Jake Graving,
  Christoph Reinders, Sarthak Yadav, Joy Banerjee, Gábor Vecsei, Adam Kraft,
  Zheng Rui, Jirka Borovec, Christian Vallentin, Semen Zhydenko, Kilian
  Pfeiffer, Ben Cook, Ismael Fernández, François-Michel De~Rainville,
  Chi-Hung Weng, Abner Ayala-Acevedo, Raphael Meudec, Matias Laporte, et~al.
\newblock {imgaug}.
\newblock \url{https://github.com/aleju/imgaug}, 2020.
\newblock Online; accessed 01-Feb-2020.

\end{thebibliography}

references.bib
% This file was created with JabRef 2.10.
% Encoding: UTF-8


@InProceedings{cciccek20163d,
  Title                    = {3D U-Net: learning dense volumetric segmentation from sparse annotation},
  Author                   = {{\c{C}}i{\c{c}}ek, {\"O}zg{\"u}n and Abdulkadir, Ahmed and Lienkamp, Soeren S and Brox, Thomas and Ronneberger, Olaf},
  Booktitle                = {International conference on medical image computing and computer-assisted intervention},
  Year                     = {2016},
  Organization             = {Springer},
  Pages                    = {424--432}
}

@InProceedings{abadi2016tensorflow,
  Title                    = {Tensorflow: A system for large-scale machine learning},
  Author                   = {Abadi, Mart{\'\i}n and Barham, Paul and Chen, Jianmin and Chen, Zhifeng and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Ghemawat, Sanjay and Irving, Geoffrey and Isard, Michael and others},
  Booktitle                = {12th $\{$USENIX$\}$ Symposium on Operating Systems Design and Implementation ($\{$OSDI$\}$ 16)},
  Year                     = {2016},
  Pages                    = {265--283}
}

@Article{avants2009advanced,
  Title                    = {Advanced normalization tools (ANTS)},
  Author                   = {Avants, Brian B and Tustison, Nick and Song, Gang},
  Journal                  = {Insight j},
  Year                     = {2009},
  Number                   = {365},
  Pages                    = {1--35},
  Volume                   = {2}
}

@Article{bauer2013survey,
  Title                    = {A survey of MRI-based medical image analysis for brain tumor studies},
  Author                   = {Bauer, Stefan and Wiest, Roland and Nolte, Lutz-P and Reyes, Mauricio},
  Journal                  = {Physics in Medicine \& Biology},
  Year                     = {2013},
  Number                   = {13},
  Pages                    = {R97},
  Volume                   = {58},

  Publisher                = {IOP Publishing}
}

@Article{binaghi2016collection,
  Title                    = {Collection and fuzzy estimation of truth labels in glial tumour segmentation studies},
  Author                   = {Binaghi, Elisabetta and Pedoia, Valentina and Balbi, Sergio},
  Journal                  = {Computer Methods in Biomechanics and Biomedical Engineering: Imaging \& Visualization},
  Year                     = {2016},
  Number                   = {3-4},
  Pages                    = {214--228},
  Volume                   = {4},

  Publisher                = {Taylor \& Francis}
}

@Article{bondy2008brain,
  Title                    = {Brain tumor epidemiology: consensus from the Brain Tumor Epidemiology Consortium},
  Author                   = {Bondy, Melissa L and Scheurer, Michael E and Malmer, Beatrice and Barnholtz-Sloan, Jill S and Davis, Faith G and Il'Yasova, Dora and Kruchko, Carol and McCarthy, Bridget J and Rajaraman, Preetha and Schwartzbaum, Judith A and others},
  Journal                  = {Cancer},
  Year                     = {2008},
  Number                   = {S7},
  Pages                    = {1953--1968},
  Volume                   = {113},

  Publisher                = {Wiley Online Library}
}

@Article{diciccio1996bootstrap,
  Title                    = {Bootstrap confidence intervals},
  Author                   = {DiCiccio, Thomas J and Efron, Bradley},
  Journal                  = {Statistical science},
  Year                     = {1996},
  Pages                    = {189--212},

  Publisher                = {JSTOR}
}

@Article{dvorak2015structured,
  Title                    = {Structured prediction with convolutional neural networks for multimodal brain tumor segmentation},
  Author                   = {Dvorak, Pavel and Menze, Bjoern},
  Journal                  = {Proceeding of the multimodal brain tumor image segmentation challenge},
  Year                     = {2015},
  Pages                    = {13--24}
}

@InCollection{efron1992bootstrap,
  Title                    = {Bootstrap methods: another look at the jackknife},
  Author                   = {Efron, Bradley},
  Booktitle                = {Breakthroughs in statistics},
  Publisher                = {Springer},
  Year                     = {1992},
  Pages                    = {569--593}
}

@Article{feng2020brain,
  Title                    = {Brain tumor segmentation using an ensemble of 3d u-nets and overall survival prediction using radiomic features},
  Author                   = {Feng, Xue and Tustison, Nicholas J and Patel, Sohil H and Meyer, Craig H},
  Journal                  = {Frontiers in Computational Neuroscience},
  Year                     = {2020},
  Pages                    = {25},
  Volume                   = {14},

  Publisher                = {Frontiers}
}

@Article{fountain2017volumetric,
  Title                    = {Volumetric growth rates of meningioma and its correlation with histological diagnosis and clinical outcome: a systematic review},
  Author                   = {Fountain, Daniel M and Soon, Wai Cheong and Matys, Tomasz and Guilfoyle, Mathew R and Kirollos, Ramez and Santarius, Thomas},
  Journal                  = {Acta neurochirurgica},
  Year                     = {2017},
  Number                   = {3},
  Pages                    = {435--445},
  Volume                   = {159},

  Publisher                = {Springer}
}

@Article{goldbrunner2016eano,
  Title                    = {EANO guidelines for the diagnosis and treatment of meningiomas},
  Author                   = {Goldbrunner, Roland and Minniti, Giuseppe and Preusser, Matthias and Jenkinson, Michael D and Sallabanda, Kita and Houdart, Emmanuel and von Deimling, Andreas and Stavrinou, Pantelis and Lefranc, Florence and Lund-Johansen, Morten and others},
  Journal                  = {The Lancet Oncology},
  Year                     = {2016},
  Number                   = {9},
  Pages                    = {e383--e391},
  Volume                   = {17},

  Publisher                = {Elsevier}
}

@Article{havaei2017brain,
  Title                    = {Brain tumor segmentation with deep neural networks},
  Author                   = {Havaei, Mohammad and Davy, Axel and Warde-Farley, David and Biard, Antoine and Courville, Aaron and Bengio, Yoshua and Pal, Chris and Jodoin, Pierre-Marc and Larochelle, Hugo},
  Journal                  = {Medical image analysis},
  Year                     = {2017},
  Pages                    = {18--31},
  Volume                   = {35},

  Publisher                = {Elsevier}
}

@Article{icsin2016review,
  Title                    = {Review of MRI-based brain tumor image segmentation using deep learning methods},
  Author                   = {I{\c{s}}{\i}n, Ali and Direko{\u{g}}lu, Cem and {\c{S}}ah, Melike},
  Journal                  = {Procedia Computer Science},
  Year                     = {2016},
  Pages                    = {317--324},
  Volume                   = {102},

  Publisher                = {Elsevier}
}

@InProceedings{isensee2018no,
  Title                    = {No new-net},
  Author                   = {Isensee, Fabian and Kickingereder, Philipp and Wick, Wolfgang and Bendszus, Martin and Maier-Hein, Klaus H},
  Booktitle                = {International MICCAI Brainlesion Workshop},
  Year                     = {2018},
  Organization             = {Springer},
  Pages                    = {234--244}
}

@Misc{imgaug,
  Title                    = {{imgaug}},

  Author                   = {Jung, Alexander B.
 and Wada, Kentaro
 and Crall, Jon
 and Tanaka, Satoshi
 and Graving, Jake
 and Reinders, Christoph
 and Yadav, Sarthak
 and Banerjee, Joy
 and Vecsei, Gábor
 and Kraft, Adam
 and Rui, Zheng
 and Borovec, Jirka
 and Vallentin, Christian
 and Zhydenko, Semen
 and Pfeiffer, Kilian
 and Cook, Ben
 and Fernández, Ismael
 and De Rainville, François-Michel
 and Weng, Chi-Hung
 and Ayala-Acevedo, Abner
 and Meudec, Raphael
 and Laporte, Matias
 and others},
  HowPublished             = {\url{https://github.com/aleju/imgaug}},
  Note                     = {Online; accessed 01-Feb-2020},
  Year                     = {2020}
}

@Article{kamnitsas2017efficient,
  Title                    = {Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation},
  Author                   = {Kamnitsas, Konstantinos and Ledig, Christian and Newcombe, Virginia FJ and Simpson, Joanna P and Kane, Andrew D and Menon, David K and Rueckert, Daniel and Glocker, Ben},
  Journal                  = {Medical image analysis},
  Year                     = {2017},
  Pages                    = {61--78},
  Volume                   = {36},

  Publisher                = {Elsevier}
}

@Article{kunimatsu2016variants,
  Title                    = {Variants of meningiomas: a review of imaging findings and clinical features},
  Author                   = {Kunimatsu, Akira and Kunimatsu, Natsuko and Kamiya, Kouhei and Katsura, Masaki and Mori, Harushi and Ohtomo, Kuni},
  Journal                  = {Japanese journal of radiology},
  Year                     = {2016},
  Number                   = {7},
  Pages                    = {459--469},
  Volume                   = {34},

  Publisher                = {Springer}
}

@Article{laukamp2020automated,
  Title                    = {Automated Meningioma Segmentation in Multiparametric MRI},
  Author                   = {Laukamp, Kai Roman and Pennig, Lenhard and Thiele, Frank and Reimer, Robert and G{\"o}rtz, Lukas and Shakirin, Georgy and Zopfs, David and Timmer, Marco and Perkuhn, Michael and Borggrefe, Jan},
  Journal                  = {Clinical Neuroradiology},
  Year                     = {2020},
  Pages                    = {1--10},

  Publisher                = {Springer}
}

@Article{laukamp2019fully,
  Title                    = {Fully automated detection and segmentation of meningiomas using deep learning on routine multiparametric MRI},
  Author                   = {Laukamp, Kai Roman and Thiele, Frank and Shakirin, Georgy and Zopfs, David and Faymonville, Andrea and Timmer, Marco and Maintz, David and Perkuhn, Michael and Borggrefe, Jan},
  Journal                  = {European radiology},
  Year                     = {2019},
  Number                   = {1},
  Pages                    = {124--132},
  Volume                   = {29},

  Publisher                = {Springer}
}

@Article{lee2019efficient,
  Title                    = {Efficient 3D Fully Convolutional Networks for Pulmonary Lobe Segmentation in CT Images},
  Author                   = {Lee, Hoileong and Matin, Tahreema and Gleeson, Fergus and Grau, Vicente},
  Journal                  = {arXiv preprint arXiv:1909.07474},
  Year                     = {2019}
}

@Article{menze2014multimodal,
  Title                    = {The multimodal brain tumor image segmentation benchmark (BRATS)},
  Author                   = {Menze, Bjoern H and Jakab, Andras and Bauer, Stefan and Kalpathy-Cramer, Jayashree and Farahani, Keyvan and Kirby, Justin and Burren, Yuliya and Porz, Nicole and Slotboom, Johannes and Wiest, Roland and others},
  Journal                  = {IEEE transactions on medical imaging},
  Year                     = {2014},
  Number                   = {10},
  Pages                    = {1993--2024},
  Volume                   = {34},

  Publisher                = {IEEE}
}

@InProceedings{myronenko20183d,
  Title                    = {3D MRI brain tumor segmentation using autoencoder regularization},
  Author                   = {Myronenko, Andriy},
  Booktitle                = {International MICCAI Brainlesion Workshop},
  Year                     = {2018},
  Organization             = {Springer},
  Pages                    = {311--320}
}

@Article{nyul2000new,
  Title                    = {New variants of a method of MRI scale standardization},
  Author                   = {Ny{\'u}l, L{\'a}szl{\'o} G and Udupa, Jayaram K and Zhang, Xuan},
  Journal                  = {IEEE transactions on medical imaging},
  Year                     = {2000},
  Number                   = {2},
  Pages                    = {143--150},
  Volume                   = {19},

  Publisher                = {IEEE}
}

@InProceedings{paszke2019pytorch,
  Title                    = {PyTorch: An imperative style, high-performance deep learning library},
  Author                   = {Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and others},
  Booktitle                = {Advances in Neural Information Processing Systems},
  Year                     = {2019},
  Pages                    = {8024--8035}
}

@Article{pereira2016brain,
  Title                    = {Brain tumor segmentation using convolutional neural networks in MRI images},
  Author                   = {Pereira, S{\'e}rgio and Pinto, Adriano and Alves, Victor and Silva, Carlos A},
  Journal                  = {IEEE transactions on medical imaging},
  Year                     = {2016},
  Number                   = {5},
  Pages                    = {1240--1251},
  Volume                   = {35},

  Publisher                = {IEEE}
}

@InProceedings{ronneberger2015u,
  Title                    = {U-net: Convolutional networks for biomedical image segmentation},
  Author                   = {Ronneberger, Olaf and Fischer, Philipp and Brox, Thomas},
  Booktitle                = {International Conference on Medical image computing and computer-assisted intervention},
  Year                     = {2015},
  Organization             = {Springer},
  Pages                    = {234--241}
}

@Article{spasic2016incidental,
  Title                    = {Incidental meningiomas: management in the neuroimaging era},
  Author                   = {Spasic, Marko and Pelargos, Panayiotis E and Barnette, Natalie and Bhatt, Nikhilesh S and Lee, Seung James and Ung, Nolan and Gopen, Quinton and Yang, Isaac},
  Journal                  = {Neurosurgery Clinics},
  Year                     = {2016},
  Number                   = {2},
  Pages                    = {229--238},
  Volume                   = {27},

  Publisher                = {Elsevier}
}

@Article{tustison2010n4itk,
  Title                    = {N4ITK: improved N3 bias correction},
  Author                   = {Tustison, Nicholas J and Avants, Brian B and Cook, Philip A and Zheng, Yuanjie and Egan, Alexander and Yushkevich, Paul A and Gee, James C},
  Journal                  = {IEEE transactions on medical imaging},
  Year                     = {2010},
  Number                   = {6},
  Pages                    = {1310--1320},
  Volume                   = {29},

  Publisher                = {IEEE}
}

@Article{ueda2019technical,
  Title                    = {Technical and clinical overview of deep learning in radiology},
  Author                   = {Ueda, Daiju and Shimazaki, Akitoshi and Miki, Yukio},
  Journal                  = {Japanese journal of radiology},
  Year                     = {2019},
  Number                   = {1},
  Pages                    = {15--33},
  Volume                   = {37},

  Publisher                = {Springer}
}

@Article{watts2014magnetic,
  Title                    = {Magnetic resonance imaging of meningiomas: a pictorial review},
  Author                   = {Watts, J and Box, G and Galvin, A and Brotchie, P and Trost, N and Sutherland, T},
  Journal                  = {Insights into imaging},
  Year                     = {2014},
  Number                   = {1},
  Pages                    = {113--122},
  Volume                   = {5},

  Publisher                = {SpringerOpen}
}

@InProceedings{xu2018multi,
  Title                    = {Multi-scale masked 3-D U-net for brain tumor segmentation},
  Author                   = {Xu, Yanwu and Gong, Mingming and Fu, Huan and Tao, Dacheng and Zhang, Kun and Batmanghelich, Kayhan},
  Booktitle                = {International MICCAI Brainlesion Workshop},
  Year                     = {2018},
  Organization             = {Springer},
  Pages                    = {222--233}
}

@Article{zhao2018deep,
  Title                    = {A deep learning model integrating FCNNs and CRFs for brain tumor segmentation},
  Author                   = {Zhao, Xiaomei and Wu, Yihong and Song, Guidong and Li, Zhenye and Zhang, Yazhuo and Fan, Yong},
  Journal                  = {Medical image analysis},
  Year                     = {2018},
  Pages                    = {98--111},
  Volume                   = {43},

  Publisher                = {Elsevier}
}

@Article{zikic2014segmentation,
  Title                    = {Segmentation of brain tumor tissues with convolutional neural networks},
  Author                   = {Zikic, Darko and Ioannou, Yani and Brown, Matthew and Criminisi, Antonio},
  Journal                  = {Proceedings MICCAI-BRATS},
  Year                     = {2014},
  Pages                    = {36--39}
}

@inproceedings{pieper20043d,
  title={3D Slicer},
  author={Pieper, Steve and Halle, Michael and Kikinis, Ron},
  booktitle={2004 2nd IEEE international symposium on biomedical imaging: nano to macro (IEEE Cat No. 04EX821)},
  pages={632--635},
  year={2004},
  organization={IEEE}
}


@article{ostrom2019cbtrus,
  title={CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2012--2016},
  author={Ostrom, Quinn T and Cioffi, Gino and Gittleman, Haley and Patil, Nirav and Waite, Kristin and Kruchko, Carol and Barnholtz-Sloan, Jill S},
  journal={Neuro-oncology},
  volume={21},
  number={Supplement\_5},
  pages={v1--v100},
  year={2019},
  publisher={Oxford University Press US}
}

@article{berntsen2020volumetric,
  title={Volumetric segmentation of glioblastoma progression compared to bidimensional products and clinical radiological reports},
  author={Berntsen, Erik Magnus and Stensj{\o}en, Anne Line and Langlo, Maren Staurset and Simonsen, Solveig Quam and Christensen, P{\aa}l and Moholdt, Viggo Andreas and Solheim, Ole},
  journal={Acta Neurochirurgica},
  volume={162},
  number={2},
  pages={379--387},
  year={2020},
  publisher={Springer}
}

@article{masters2018revisiting,
  title={Revisiting small batch training for deep neural networks},
  author={Masters, Dominic and Luschi, Carlo},
  journal={arXiv preprint arXiv:1804.07612},
  year={2018}
}

@article{killeen2005alternative,
  title={An alternative to null-hypothesis significance tests},
  author={Killeen, Peter R},
  journal={Psychological science},
  volume={16},
  number={5},
  pages={345--353},
  year={2005},
  publisher={SAGE Publications Sage CA: Los Angeles, CA}
}



@article{KeskarMNST16,
  author    = {Nitish Shirish Keskar and
               Dheevatsa Mudigere and
               Jorge Nocedal and
               Mikhail Smelyanskiy and
               Ping Tak Peter Tang},
  title     = {On Large-Batch Training for Deep Learning: Generalization Gap and
               Sharp Minima},
  journal   = {CoRR},
  volume    = {abs/1609.04836},
  year      = {2016},
  url       = {http://arxiv.org/abs/1609.04836},
  archivePrefix = {arXiv},
  eprint    = {1609.04836},
  timestamp = {Mon, 13 Aug 2018 16:46:48 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/KeskarMNST16.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

@incollection{zhou2018unet++,
  title={Unet++: A nested u-net architecture for medical image segmentation},
  author={Zhou, Zongwei and Siddiquee, Md Mahfuzur Rahman and Tajbakhsh, Nima and Liang, Jianming},
  booktitle={Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support},
  pages={3--11},
  year={2018},
  publisher={Springer}
}

@article{alom2018recurrent,
  title={Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation},
  author={Alom, Md Zahangir and Hasan, Mahmudul and Yakopcic, Chris and Taha, Tarek M and Asari, Vijayan K},
  journal={arXiv preprint arXiv:1802.06955},
  year={2018}
}

@article{isensee2018nnu,
  title={nnu-net: Self-adapting framework for u-net-based medical image segmentation},
  author={Isensee, Fabian and Petersen, Jens and Klein, Andre and Zimmerer, David and Jaeger, Paul F and Kohl, Simon and Wasserthal, Jakob and Koehler, Gregor and Norajitra, Tobias and Wirkert, Sebastian and others},
  journal={arXiv preprint arXiv:1809.10486},
  year={2018}
}

@article{avants2011reproducible,
  title={A reproducible evaluation of ANTs similarity metric performance in brain image registration},
  author={Avants, Brian B and Tustison, Nicholas J and Song, Gang and Cook, Philip A and Klein, Arno and Gee, James C},
  journal={Neuroimage},
  volume={54},
  number={3},
  pages={2033--2044},
  year={2011},
  publisher={Elsevier}
}

@article{niu2020hmanet,
  title={Hmanet: Hybrid multiple attention network for semantic segmentation in aerial images},
  author={Niu, Ruigang},
  journal={arXiv preprint arXiv:2001.02870},
  year={2020}
}

@inproceedings{fu2019dual,
  title={Dual attention network for scene segmentation},
  author={Fu, Jun and Liu, Jing and Tian, Haijie and Li, Yong and Bao, Yongjun and Fang, Zhiwei and Lu, Hanqing},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={3146--3154},
  year={2019}
}

@article{cheng2020fully,
  title={Fully Convolutional Attention Network for Biomedical Image Segmentation},
  author={Cheng, Junlong and Tian, Shengwei and Yu, Long and Lu, Hongchun and Lv, Xiaoyi},
  journal={Artificial Intelligence in Medicine},
  pages={101899},
  year={2020},
  publisher={Elsevier}
}

@inproceedings{mou2019cs,
  title={CS-Net: channel and spatial attention network for curvilinear structure segmentation},
  author={Mou, Lei and Zhao, Yitian and Chen, Li and Cheng, Jun and Gu, Zaiwang and Hao, Huaying and Qi, Hong and Zheng, Yalin and Frangi, Alejandro and Liu, Jiang},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages={721--730},
  year={2019},
  organization={Springer}
}

@article{wang2020multi,
  title={Multi-resolution convolutional neural networks for inverse problems},
  author={Wang, Feng and Eljarrat, Alberto and M{\"u}ller, Johannes and Henninen, Trond R and Erni, Rolf and Koch, Christoph T},
  journal={Scientific reports},
  volume={10},
  number={1},
  pages={1--11},
  year={2020},
  publisher={Nature Publishing Group}
}

@inproceedings{abraham2019novel,
  title={A novel focal tversky loss function with improved attention u-net for lesion segmentation},
  author={Abraham, Nabila and Khan, Naimul Mefraz},
  booktitle={2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)},
  pages={683--687},
  year={2019},
  organization={IEEE}
}

@article{sang2020pcanet,
  title={PCANet: Pyramid convolutional attention network for semantic segmentation},
  author={Sang, Haiwei and Zhou, Qiuhao and Zhao, Yong},
  journal={Image and Vision Computing},
  volume={103},
  pages={103997},
  year={2020},
  publisher={Elsevier}
}

@article{kavur2020chaos,
  title={CHAOS Challenge--Combined (CT-MR) Healthy Abdominal Organ Segmentation},
  author={Kavur, A Emre and Gezer, N Sinem and Bar{\i}{\c{s}}, Mustafa and Conze, Pierre-Henri and Groza, Vladimir and Pham, Duc Duy and Chatterjee, Soumick and Ernst, Philipp and {\"O}zkan, Sava{\c{s}} and Baydar, Bora and others},
  journal={arXiv preprint arXiv:2001.06535},
  year={2020}
}

@misc{simpson2019large,
      title={A large annotated medical image dataset for the development and evaluation of segmentation algorithms}, 
      author={Amber L. Simpson and Michela Antonelli and Spyridon Bakas and Michel Bilello and Keyvan Farahani and Bram van Ginneken and Annette Kopp-Schneider and Bennett A. Landman and Geert Litjens and Bjoern Menze and Olaf Ronneberger and Ronald M. Summers and Patrick Bilic and Patrick F. Christ and Richard K. G. Do and Marc Gollub and Jennifer Golia-Pernicka and Stephan H. Heckers and William R. Jarnagin and Maureen K. McHugo and Sandy Napel and Eugene Vorontsov and Lena Maier-Hein and M. Jorge Cardoso},
      year={2019},
      eprint={1902.09063},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@article{vesal20192d,
  title={A 2D dilated residual U-Net for multi-organ segmentation in thoracic CT},
  author={Vesal, Sulaiman and Ravikumar, Nishant and Maier, Andreas},
  journal={arXiv preprint arXiv:1905.07710},
  year={2019}
}

@inproceedings{long2015fully,
  title={Fully convolutional networks for semantic segmentation},
  author={Long, Jonathan and Shelhamer, Evan and Darrell, Trevor},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={3431--3440},
  year={2015}
}

@inproceedings{zhou2017fixed,
  title={A fixed-point model for pancreas segmentation in abdominal CT scans},
  author={Zhou, Yuyin and Xie, Lingxi and Shen, Wei and Wang, Yan and Fishman, Elliot K and Yuille, Alan L},
  booktitle={International conference on medical image computing and computer-assisted intervention},
  pages={693--701},
  year={2017},
  organization={Springer}
}

@article{wang2019abdominal,
  title={Abdominal multi-organ segmentation with organ-attention networks and statistical fusion},
  author={Wang, Yan and Zhou, Yuyin and Shen, Wei and Park, Seyoun and Fishman, Elliot K and Yuille, Alan L},
  journal={Medical image analysis},
  volume={55},
  pages={88--102},
  year={2019},
  publisher={Elsevier}
}

@article{zhuang2016multi,
  title={Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI},
  author={Zhuang, Xiahai and Shen, Juan},
  journal={Medical image analysis},
  volume={31},
  pages={77--87},
  year={2016},
  publisher={Elsevier}
}

@article{setio2016pulmonary,
  title={Pulmonary nodule detection in CT images: false positive reduction using multi-view convolutional networks},
  author={Setio, Arnaud Arindra Adiyoso and Ciompi, Francesco and Litjens, Geert and Gerke, Paul and Jacobs, Colin and Van Riel, Sarah J and Wille, Mathilde Marie Winkler and Naqibullah, Matiullah and S{\'a}nchez, Clara I and van Ginneken, Bram},
  journal={IEEE transactions on medical imaging},
  volume={35},
  number={5},
  pages={1160--1169},
  year={2016},
  publisher={IEEE}
}

@article{li2020deep,
  title={Deep Reinforcement Learning for Weakly-Supervised Lymph Node Segmentation in CT Images},
  author={Li, Zhe and Xia, Yong},
  journal={IEEE Journal of Biomedical and Health Informatics},
  year={2020},
  publisher={IEEE}
}

@article{liu2016mediastinal,
  title={Mediastinal lymph node detection and station mapping on chest CT using spatial priors and random forest},
  author={Liu, Jiamin and Hoffman, Joanne and Zhao, Jocelyn and Yao, Jianhua and Lu, Le and Kim, Lauren and Turkbey, Evrim B and Summers, Ronald M},
  journal={Medical physics},
  volume={43},
  number={7},
  pages={4362--4374},
  year={2016},
  publisher={Wiley Online Library}
}

@article{oda2017automated,
  title={Automated mediastinal lymph node detection from CT volumes based on intensity targeted radial structure tensor analysis},
  author={Oda, Hirohisa and Bhatia, Kanwal K and Oda, Masahiro and Kitasaka, Takayuki and Iwano, Shingo and Homma, Hirotoshi and Takabatake, Hirotsugu and Mori, Masaki and Natori, Hiroshi and Schnabel, Julia A and others},
  journal={Journal of Medical Imaging},
  volume={4},
  number={4},
  pages={044502},
  year={2017},
  publisher={International Society for Optics and Photonics}
}

@inproceedings{nogues2016automatic,
  title={Automatic lymph node cluster segmentation using holistically-nested neural networks and structured optimization in CT images},
  author={Nogues, Isabella and Lu, Le and Wang, Xiaosong and Roth, Holger and Bertasius, Gedas and Lay, Nathan and Shi, Jianbo and Tsehay, Yohannes and Summers, Ronald M},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages={388--397},
  year={2016},
  organization={Springer}
}

@inproceedings{zhu2020lymph,
  title={Lymph Node Gross Tumor Volume Detection and Segmentation via Distance-Based Gating Using 3D CT/PET Imaging in Radiotherapy},
  author={Zhu, Zhuotun and Jin, Dakai and Yan, Ke and Ho, Tsung-Ying and Ye, Xianghua and Guo, Dazhou and Chao, Chun-Hung and Xiao, Jing and Yuille, Alan and Lu, Le},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages={753--762},
  year={2020},
  organization={Springer}
}

@inproceedings{xu2020focal,
  title={Focal Loss Function based DeepLabv3+ for Pathological Lymph Node Segmentation on PET/CT},
  author={Xu, Guoping and Cao, Hanqiang and Dong, Youli and Yue, Chunyi and Li, Kexin and Tong, Yubing},
  booktitle={Proceedings of the 2020 2nd International Conference on Intelligent Medicine and Image Processing},
  pages={24--28},
  year={2020}
}

@article{bouget2019semantic,
  title={Semantic segmentation and detection of mediastinal lymph nodes and anatomical structures in CT data for lung cancer staging},
  author={Bouget, David and J{\o}rgensen, Arve and Kiss, Gabriel and Leira, Haakon Olav and Lang{\o}, Thomas},
  journal={International journal of computer assisted radiology and surgery},
  volume={14},
  number={6},
  pages={977--986},
  year={2019},
  publisher={Springer}
}

@inproceedings{paing2019automatic,
  title={Automatic Detection of Mediastinal Lymph Nodes using 3D Convolutional Neural Network},
  author={Paing, May Phu and Pintavirooj, Chuchart and Tungjitkusolmun, Supan and Win, Kyi Pyar and Hamamoto, Kazuhiko},
  booktitle={Proceedings of the 2019 4th International Conference on Biomedical Imaging, Signal Processing},
  pages={26--31},
  year={2019}
}

@article{el2014international,
  title={International association for the study of lung cancer (IASLC) lymph node map: radiologic review with CT illustration},
  author={El-Sherief, Ahmed H and Lau, Charles T and Wu, Carol C and Drake, Richard L and Abbott, Gerald F and Rice, Thomas W},
  journal={Radiographics},
  volume={34},
  number={6},
  pages={1680--1691},
  year={2014},
  publisher={Radiological Society of North America}
}

@article{eisenhauer2009new,
  title={New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1)},
  author={Eisenhauer, Elizabeth A and Therasse, Patrick and Bogaerts, Jan and Schwartz, Lawrence H and Sargent, D and Ford, Robert and Dancey, Janet and Arbuck, S and Gwyther, Steve and Mooney, Margaret and others},
  journal={European journal of cancer},
  volume={45},
  number={2},
  pages={228--247},
  year={2009},
  publisher={Elsevier}
}

@article{hofmanninger2020automatic,
  title={Automatic lung segmentation in routine imaging is a data diversity problem, not a methodology problem},
  author={Hofmanninger, Johannes and Prayer, Florian and Pan, Jeanny and Rohrich, Sebastian and Prosch, Helmut and Langs, Georg},
  journal={arXiv preprint arXiv:2001.11767},
  year={2020}
}

@inproceedings{roth2014new,
  title={A new 2.5 D representation for lymph node detection using random sets of deep convolutional neural network observations},
  author={Roth, Holger R and Lu, Le and Seff, Ari and Cherry, Kevin M and Hoffman, Joanne and Wang, Shijun and Liu, Jiamin and Turkbey, Evrim and Summers, Ronald M},
  booktitle={International conference on medical image computing and computer-assisted intervention},
  pages={520--527},
  year={2014},
  organization={Springer}
}

@inproceedings{he2017mask,
  title={Mask r-cnn},
  author={He, Kaiming and Gkioxari, Georgia and Doll{\'a}r, Piotr and Girshick, Ross},
  booktitle={Proceedings of the IEEE international conference on computer vision},
  pages={2961--2969},
  year={2017}
}
@article{redmon2018yolov3,
  title={Yolov3: An incremental improvement},
  author={Redmon, Joseph and Farhadi, Ali},
  journal={arXiv preprint arXiv:1804.02767},
  year={2018}
}

@inproceedings{liu2014mediastinal,
  title={Mediastinal lymph node detection on thoracic CT scans using spatial prior from multi-atlas label fusion},
  author={Liu, Jiamin and Zhao, Jocelyn and Hoffman, Joanne and Yao, Jianhua and Zhang, Weidong and Turkbey, Evrim B and Wang, Shijun and Kim, Christine and Summers, Ronald M},
  booktitle={Medical Imaging 2014: Computer-Aided Diagnosis},
  volume={9035},
  pages={90350M},
  year={2014},
  organization={International Society for Optics and Photonics}
}

@inproceedings{oda2017hessian,
  title={Hessian-assisted supervoxel: structure-oriented voxel clustering and application to mediastinal lymph node detection from CT volumes},
  author={Oda, Hirohisa and Bhatia, Kanwal K and Oda, Masahiro and Kitasaka, Takayuki and Iwano, Shingo and Homma, Hirotoshi and Takabatake, Hirotsugu and Mori, Masaki and Natori, Hiroshi and Schnabel, Julia A and others},
  booktitle={Medical Imaging 2017: Computer-Aided Diagnosis},
  volume={10134},
  pages={101341D},
  year={2017},
  organization={International Society for Optics and Photonics}
}

@inproceedings{oda2018dense,
  title={Dense volumetric detection and segmentation of mediastinal lymph nodes in chest CT images},
  author={Oda, Hirohisa and Roth, Holger R and Bhatia, Kanwal K and Oda, Masahiro and Kitasaka, Takayuki and Iwano, Shingo and Homma, Hirotoshi and Takabatake, Hirotsugu and Mori, Masaki and Natori, Hiroshi and others},
  booktitle={Medical Imaging 2018: Computer-Aided Diagnosis},
  volume={10575},
  pages={1057502},
  year={2018},
  organization={International Society for Optics and Photonics}
}

@inproceedings{xu2018thoracic,
  title={Thoracic lymph node station recognition on CT images based on automatic anatomy recognition with an optimal parent strategy},
  author={Xu, Guoping and Udupa, Jayaram K and Tong, Yubing and Cao, Hanqiang and Odhner, Dewey and Torigian, Drew A and Wu, Xingyu},
  booktitle={Medical Imaging 2018: Image Processing},
  volume={10574},
  pages={105742F},
  year={2018},
  organization={International Society for Optics and Photonics}
}

@inproceedings{redmon2016you,
  title={You only look once: Unified, real-time object detection},
  author={Redmon, Joseph and Divvala, Santosh and Girshick, Ross and Farhadi, Ali},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={779--788},
  year={2016}
}

@inproceedings{he2016deep,
  title={Deep residual learning for image recognition},
  author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  pages={770--778},
  year={2016}
}

@article{bai2017human,
  title={Human-level cmr image analysis with deep fully convolutional networks},
  author={Bai, Wenjia and Sinclair, Matthew and Tarroni, Giacomo and Oktay, Ozan and Rajchl, Martin and Vaillant, Ghislain and Lee, Aaron M and Aung, Nay and Lukaschuk, Elena and Sanghvi, Mihir M and others},
  year={2017}
}

@article{liao2019evaluate,
  title={Evaluate the malignancy of pulmonary nodules using the 3-d deep leaky noisy-or network},
  author={Liao, Fangzhou and Liang, Ming and Li, Zhe and Hu, Xiaolin and Song, Sen},
  journal={IEEE transactions on neural networks and learning systems},
  volume={30},
  number={11},
  pages={3484--3495},
  year={2019},
  publisher={IEEE}
}

@misc{jetley2018learn,
      title={Learn To Pay Attention}, 
      author={Saumya Jetley and Nicholas A. Lord and Namhoon Lee and Philip H. S. Torr},
      year={2018},
      eprint={1804.02391},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

@article{chen2017rethinking,
  title={Rethinking atrous convolution for semantic image segmentation},
  author={Chen, Liang-Chieh and Papandreou, George and Schroff, Florian and Adam, Hartwig},
  journal={arXiv preprint arXiv:1706.05587},
  year={2017}
}

@article{sinha2020multi,
  title={Multi-scale self-guided attention for medical image segmentation},
  author={Sinha, Ashish and Dolz, Jose},
  journal={IEEE Journal of Biomedical and Health Informatics},
  year={2020},
  publisher={IEEE}
}

@inproceedings{lee2015deeply,
  title={Deeply-supervised nets},
  author={Lee, Chen-Yu and Xie, Saining and Gallagher, Patrick and Zhang, Zhengyou and Tu, Zhuowen},
  booktitle={Artificial intelligence and statistics},
  pages={562--570},
  year={2015}
}

@Article{falk2010,
  title={Chapter 1},
  author={Falk S, Williams C},
  journal={Lung Cancer: the facts (3rd ed.)},
  pages={pp3-4},
  year={2010}
}

@article{schwartz2009evaluation,
  title={Evaluation of lymph nodes with RECIST 1.1},
  author={Schwartz, LH and Bogaerts, J and Ford, R and Shankar, L and Therasse, P and Gwyther, S and Eisenhauer, EA},
  journal={European journal of cancer},
  volume={45},
  number={2},
  pages={261--267},
  year={2009},
  publisher={Elsevier}
}

@article{sorger2017multimodal,
  title={A multimodal image guiding system for Navigated Ultrasound Bronchoscopy (EBUS): A human feasibility study},
  author={Sorger, Hanne and Hofstad, Erlend Fagertun and Amundsen, Tore and Lang{\o}, Thomas and Bakeng, Janne Beate Lervik and Leira, H{\aa}kon Olav},
  journal={PloS one},
  volume={12},
  number={2},
  pages={e0171841},
  year={2017},
  publisher={Public Library of Science San Francisco, CA USA}
}

@article{mcerlean2013intra,
  title={Intra-and interobserver variability in CT measurements in oncology},
  author={McErlean, Aoife and Panicek, David M and Zabor, Emily C and Moskowitz, Chaya S and Bitar, Richard and Motzer, Robert J and Hricak, Hedvig and Ginsberg, Michelle S},
  journal={Radiology},
  volume={269},
  number={2},
  pages={451--459},
  year={2013},
  publisher={Radiological Society of North America}
}

@inproceedings{kamnitsas2016deepmedic,
  title={DeepMedic for brain tumor segmentation},
  author={Kamnitsas, Konstantinos and Ferrante, Enzo and Parisot, Sarah and Ledig, Christian and Nori, Aditya V and Criminisi, Antonio and Rueckert, Daniel and Glocker, Ben},
  booktitle={International workshop on Brainlesion: Glioma, multiple sclerosis, stroke and traumatic brain injuries},
  pages={138--149},
  year={2016},
  organization={Springer}
}

@article{bouget2021meningioma,
  title={Meningioma segmentation in T1-weighted MRI leveraging global context and attention mechanisms},
  author={Bouget, David and Pedersen, Andr{\'e} and Hosainey, Sayied Abdol Mohieb and Solheim, Ole and Reinertsen, Ingerid},
  journal={arXiv preprint arXiv:2101.07715},
  year={2021}
}


@article{reynisson2015airway,
  title={Airway segmentation and centerline extraction from thoracic CT--comparison of a new method to state of the art commercialized methods},
  author={Reynisson, Pall Jens and Scali, Marta and Smistad, Erik and Hofstad, Erlend Fagertun and Leira, H{\aa}kon Olav and Lindseth, Frank and Nagelhus Hernes, Toril Anita and Amundsen, Tore and Sorger, Hanne and Lang{\o}, Thomas},
  journal={PloS one},
  volume={10},
  number={12},
  pages={e0144282},
  year={2015},
  publisher={Public Library of Science San Francisco, CA USA}
}

@inproceedings{bolya2019yolact,
  title={Yolact: Real-time instance segmentation},
  author={Bolya, Daniel and Zhou, Chong and Xiao, Fanyi and Lee, Yong Jae},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={9157--9166},
  year={2019}
}


@comment{jabref-meta: databaseType:bibtex;}
