肝脏分割 Using Cascaded Fully Convolutional Neural Networks and 3D Conditional Random Fields

Automatic Liver and Lesion Segmentation in CT

 Using Cascaded Fully Convolutional Neural

Networks and 3D Conditional Random Fields

Abstract. Automatic segmentation of the liver and its lesion is an important step towards deriving quantitative biomarkers for accurate clinical diagnosis and computer-aided decision support systems. This paper presents a method to automatically segment liver and lesions in CT abdomen images using cascaded fully convolutional neural networks
(CFCNs) and dense 3D conditional random fields (CRFs). We train and cascade two FCNs for a combined segmentation of the liver and its lesions. In the first step, we train a FCN to segment the liver as ROI input for a second FCN. The second FCN solely segments lesions from the predicted liver ROIs of step 1. We refine the segmentations of the CFCN using a dense 3D CRF that accounts for both spatial coherence and appearance. CFCN models were trained in a 2-fold cross-validation on the abdominal CT dataset 3DIRCAD comprising 15 hepatic tumor volumes. Our results show that CFCN-based semantic liver and lesion segmentation achieves Dice scores over 94% for liver with computation times below 100s per volume. We experimentally demonstrate the robustness of the proposed method as a decision support system with a
high accuracy and speed for usage in daily clinical routine.

Keywords: Liver, Lesion, Segmentation, FCN, CRF, CFCN, Deep Learning

摘要 肝脏及其病变的自动分割是获得准确临床诊断和计算机辅助决策支持系统的定量生物标志物的重要一步。本文提出了一种利用级联完全卷积神经网络自动分割CT腹部图像中肝脏和病灶的方法CFCN）和密集的3D条件随机场（CRF）。我们训练和级联两个FCNs，用于肝脏及其病变的组合分割。在第一步中，我们训练FCN将肝脏分割为第二个FCN的ROI输入。第二个FCN仅从第一步的预测肝脏ROI中分割出病灶。我们使用密集的3D CRF来优化CFCN的分割，同时考虑空间相干性和外观。对包含15个肝肿瘤体积的腹部CT数据集3DIRCAD进行CFCN模型的2倍交叉验证训练。我们的研究结果显示，基于CFCN的语义肝脏和病灶分割对于肝脏的骰子分数达到94％以上，计算时间低于每体积100s。我们通过实验证明了所提出方法作为一个决策支持系统的稳健性在日常临床常规中使用的准确性和速度都很高。

1Introduction

Anomalies in the shape and texture of the liver and visible lesions in CT are important biomarkers for disease progression in primary and secondary hepatic tumor disease [9]. In clinical routine, manual or semi-manual techniques are applied. These, however, are subjective, operator-dependent and very timeconsuming. In order to improve the productivity of radiologists, computer-aided methods have been developed in the past, but the challenges in automatic segmentation of combined liver and lesion remain, such as low-contrast between liver and lesion, different types of contrast levels (hyper-/hypo-intense tumors), abnormalities in tissues (metastasectomie), size and varying amount of lesions.

Nevertheless, several interactive and automatic methods have been developed to segment the liver and liver lesions in CT volumes. In 2007 and 2008,two Grand Challenges benchmarks on liver and liver lesion segmentation have been conducted [9,4]. Methods presented at the challenges were mostly based on statistical shape models. Furthermore, grey level and texture based methods have been developed [9]. Recent work on liver and lesion segmentation employs
graph cut and level set techniques [16,15,17], sigmoid edge modeling [5] or manifold and machine learning [11,6]. However, these methods are not widely applied in clinics, due to their speed and robustness on heterogeneous, low-contrast reallife CT data. Hence, interactive methods were still developed [7,1] to overcome these weaknesses, which yet involve user interaction. Deep Convolutional Neural Networks CNN have gained new attention in the scientific community for solving computer vision tasks such as object recognition, classification and segmentation [14,18], often out-competing state-of-the art methods. Most importantly, CNN methods have proven to be highly robust to varying image appearance, which motivates us to apply them to fully automatic liver and lesions segmentation in CT volumes.
Semantic image segmentation methods based on fully convolutional neural networks FCN were developed in [18], with impressive results in natural image segmentation competitions [3,24]. Likewise, new segmentation methods based on

CNN and FCNs were developed for medical image analysis, with highly competitive results compared to state-of-the-art. [20,8,23,21,19,12]

In this work, we demonstrate the combined automatic segmentation of the liver and its lesions in low-contrast heterogeneous CT volumes. Our contributions are three-fold. First, we train and apply fully convolutional CNN on CT volumes of the liver for the first time, demonstrating the adaptability to challenging segmentation of hepatic liver lesions. Second, we propose to use a cascaded fully convolutional neural network (CFCN) on CT slices, which segments liver and
lesions sequentially, leading to significantly higher segmentation quality. Third,we propose to combine the cascaded CNN in 2D with a 3D dense conditional random field approach (3DCRF) as a post-processing step, to achieve higher segmentation accuracy while preserving low computational cost and memory consumption. In the following sections, we will describe our proposed pipeline(Section 2.2) including CFCN (Section 2.3) and 3D CRF (Section 2.4), illustrate

experiments on the 3DIRCADb dataset (Section 2) and summarize the results (Section 4).Automatic Liver and Lesion Segmentation in CT using CFCNs and 3DCRFs

在这项工作中，我们展示了肝脏及其病变在低对比度异质CT体积中的自动分割。我们的贡献是三倍。首先，我们首次对肝脏的CT体积进行完全卷积CNN的训练和应用，证明了其对肝脏肝脏病变的挑战性分割的适应性。其次，我们建议在CT切片上使用级联完全卷积神经网络（CFCN），其将肝脏和肝脏分段病灶顺序，导致显着更高的分割质量。第三，我们建议将2D级联CNN和3D稠密条件随机场方法（3DCRF）作为后处理步骤进行组合，以获得更高的分割精度，同时保持低计算成本和内存消耗。在下面的章节中，我们将描述包括CFCN（2.3节）和3D CRF（2.4节）在内的我们提出的管道（2.2节），举例说明在3DIRCADb数据集上进行实验（第2节）并总结结果（第4节）。使用CFCNs和3DCRFs对CT进行自动肝脏和病灶分割

2 方法

F1 Automatic liver and lesion segmentation with cascaded fully convolutional networks (CFCN) and dense conditional random fields (CRF). Green depicts correctly predicted liver segmentation, yellow for liver false negative and false positive pixels (allwrong predictions), blue shows correctly predicted lesion segmentation and red lesion false negative and false positive pixels (all wrong predictions). In the first row, the false positive lesion prediction in B of a single UNet as proposed by [20] were eliminated in C by CFCN as a result of restricting lesion segmentation to the liver ROI region.In the second row, applying the 3DCRF to CFCN in F increases both liver and lesion segmentation accuracy further, resulting in a lesion Dice score of 82.3%

In the following section, we denote the 3D image volume as I, the total number of voxels as N and the set of possible labels as L = {0, 1, . . . , l}. For each voxel i, we define a variable x i ∈ L that denotes the assigned label. The probability of a voxel i belonging to label k given the image I is described by P (x i = k|I) and will be modelled by the FCN. In our particular study, we use L = {0, 1, 2} for background, liver and lesion, respectively.

在下面的章节中，我们将三维图像体积表示为I，将体素的总数量表示为N，将可能的标记集合表示为L = {0,1，...。。。 ，l}。 对于每个体素I，我们定义一个变量x i∈L表示分配的标签。 给定图像I的属于标签k的体素i的概率由P（x i = k | I）描述并且将由FCN建模。 在我们的特定研究中，我们分别使用L = {0,1,2}作为背景，肝脏和病变。

2.1 数据 The dataset is available on http://ircad.fr/research/3d-ircadb-01

2.2
Data preparation, processing and pipeline Pre-processing was carried out in a slice-wise fashion. First, the Hounsfield unit
values were windowed in the range [−100, 400] to exclude irrelevant organs and objects, then we increased contrast through histogram equalization. As in [20],to teach the network the desired invariance properties, we augmented the data by applying translation, rotation and addition of gaussian noise. Thereby resulting in an increased training dataset of 22,693 image slices, which were used to train two cascaded FCNs based on the UNet architecture [20]. The predicted segmentations are then refined using dense 3D Conditional Random Fields. The entire pipeline is depicted in Figure 2.

数据准备，处理和流水线预处理以分片方式进行。 首先，亨斯菲尔德单位值在[-100,400]范围内加窗以排除不相关的器官和物体，然后通过直方图均衡增加对比度。 如[20]中所述，为了向网络传递期望的不变性，我们通过应用高斯噪声的平移，旋转和相加来增加数据。 从而导致增加了22,693个图像切片的训练数据集，用于训练基于UNet架构的两个级联FCN [20]。 预测然后使用密集的3D条件随机场对细分进行细化。 整个管道如图2所示。

2.3 Cascaded Fully Convolutional Neural Networks (CFCN)

        We used the UNet architecture [20] to compute the soft label probability maps P (x i |I). The UNet architecture enables accurate pixel-wise prediction by combining spatial and contextual information in a network architecture comprising 19 convolutional layers. In our method, we trained one network to segment the liver in abdomen slices (step 1), and another network to segment the lesions,given an image of the liver (step 2). The segmented liver from step 1 is cropped and resampled to the required input size for the cascaded UNet in step 2, which further segments the lesions.

        The motivation behind the cascade approach is that it has been shown that UNets and other forms of CNNs learn a hierarchical representation of the provided data. The stacked layers of convolutional filters are tailored towards the desired classification in a data-driven manner, as opposed to designing handcrafted features for separation of different tissue types. By cascading two UNets,we ensure that the UNet in step 1 learns filters that are specific for the detection and segmentation of the liver from an overall abdominal CT scan, while the UNet in step 2 arranges a set of filters for separation of lesions from the liver tissue. Furthermore, the liver ROI helps in reducing false positives for lesions

        我们使用UNet架构[20]来计算软标签概率图P（x i | I）。 UNet体系结构通过在网络体系结构中结合空间和上下文信息来实现准确的按像素预测19个卷积层。在我们的方法中，我们训练了一个网络来分割腹部切片中的肝脏（步骤1），另一个网络分割病变，给出肝脏图像（步骤2）。来自步骤1的分段肝脏被切下并在步骤2中重新采样到级联UNet的所需输入大小，这进一步分割病变。   

        级联方法背后的动机是，已经表明UNets和其他形式的CNN学习了所提供数据的分层表示。卷积滤波器的堆叠层是针对这个问题量身定制的以数据驱动的方式进行所需的分类，而不是设计用于分离不同组织类型的手工特征。通过级联两个UNets，我们确保UNet在步骤1中学习专门用于检测的过滤器以及从整体腹部CT扫描中分离肝脏，而UNet在步骤2中安排了一组用于从肝组织分离病变的过滤器。此外，肝脏ROI有助于减少病变的假阳性

        A crucial step in training FCNs is appropriate class balancing according to the pixel-wise frequency of each class in the data. In contrast to [18], we observed that training the network to segment small structures such as lesions is not possible without class balancing, due to the high class imbalance. Therefore we introduced an additional weighting factor ω class in the cross entropy lossAutomatic Liver and Lesion Segmentation in CT using CFCNs and 3DCRFs

        训练FCNs的关键步骤是根据数据中每个类的像素频率进行适当的类平衡。 与[18]相反，我们观察到，由于类别失衡严重，如果没有班级平衡，训练网络来分割小结构（如病变）是不可能的。 因此，我们在交叉熵损失中引入了一个额外的加权因子ω类，使用CFCNs和3DCRFs在CT中进行自动肝脏和病灶分割

该论文使用3dcrf对最后的分割结果进行优化极大地提高了分割的准确率