基于深度学习的病理_组织病理学的深度学习（第二部分）

基于深度学习的病理

计算机视觉/深度学习/医学影像 (COMPUTER VISION/ DEEP LEARNING/ MEDICAL IMAGING)

In the last part, we started an introductory discussion on the present state of Deep Learning in histopathology, we discussed Histopathology, Digital Histopathology, the possibilities of Machine Learning in the area, the various applications, followed by a detailed discussion of the challenges involved in working with Digital Microscopic Slide Images and in the application of Deep Learning Algorithms to them.

在最后一部分中 ，我们开始就组织病理学中深度学习的现状进行介绍性讨论，讨论了组织病理学，数字组织病理学，该领域机器学习的可能性，各种应用程序，然后详细讨论了其中涉及的挑战与数字显微幻灯片图像配合使用，并在它们中应用深度学习算法。

In this blog, we shall be discussing in greater detail the applicability of Deep Learning to Histopathology from a methodological perspective along with the tasks it helps accomplish using relevant work for illustration.

在这个博客中，我们将从方法论的角度更详细地讨论深度学习在组织病理学中的适用性，以及使用相关工作进行说明有助于完成深度学习的任务。

The applicability of deep learning can be studied in terms of the tasks it performs or in terms of the learning paradigm, which is the classification we shall be using in this writeup. The different learning algorithms, viz a viz Deep Learning for histopathology, along with the tasks are visualized in the following overview.

可以根据深度学习执行的任务或学习范式来研究深度学习的适用性，这是我们在本文中将使用的分类。 在下面的概述中将可视化不同的学习算法，即组织病理学的深度学习，以及任务。

Based on these, a number of DL models have been proposed in the literature that are traditionally based on convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), auto-encoders(AEs) and other variants.

基于这些，在文献中已经提出了许多基于卷积神经网络(CNN)，递归神经网络(RNN)，生成对抗网络(GAN)，自动编码器(AE)和其他变体的DL模型。 。

监督学习 (Supervised Learning)

Among the supervised learning techniques, we identify three major canonical deep learning models based on the nature of tasks that are solved in digital histopathology: Classification, Regression, and Segmentation.

在有监督的学习技术中，我们根据在数字组织病理学中解决的任务的性质，确定三种主要的规范深度学习模型：分类，回归和分段。

监督分类 (Supervised Classification)

It can be further subdivided into local and global level classification. Local level classification entails identifying cells or nuclei in patches of the whole slide image. Deep Learning has proven extensively successful in pixel-wise prediction by sliding window approach over image patches that are annotated by pathologists as regions containing objects of interest( cells/nuclei) or background.One of the most prominent works in local classification came in 2019 when Qaiser et al¹ in their paper used Persistent Homology Profiles as distinguishing features in order to segment colon tumor regions by classifying patches as tumor regions or normal ones. Persistent Homology profiles are compact mathematical feature representations of a region that are distinctive as well as robust to scale, perturbations in input data, dimension, and coordinates.They used PHP of training dataset in combination with features extracted using CNN and then employed Random Forest regressions on them separately followed by a multi-stage ensemble strategy for the final classification. This hybrid approach proved to be both accurate and highly efficient wrt inference speed.In global level classification, most of the published work focusses on a patch-based classification approach for whole-slide level disease prediction task. It can involve both patch level localization as well as whole slide level classification or grading of disease. The main disadvantage of these methods is the relatively long computational time required to carry out a dense patch-wise prediction over an entire WSI. Different works have approached this problem in different ways, some using heuristic sampling strategies to more recent ones using task-driven visual attention based coarse processing.

它可以进一步细分为本地和全局级别分类。 局部水平分类需要识别整个玻片图像的斑块中的细胞或核。 深度学习已证明，通过在图像切片上滑动窗口方法在病理学家注释为包含感兴趣对象(细胞/细胞核)或背景的区域上进行滑动窗口方法，在像素方向预测方面取得了广泛的成功.2019年，当局部分类中最杰出的作品之一是Qaiser等 ¹在他们的论文中使用持久性的同源性概况如由分类补丁肿瘤区域或正常者为了区分功能段结肠肿瘤区域。 持久同源性轮廓是一个区域的紧凑数学特征表示，具有独特性，规模鲁棒性，输入数据，尺寸和坐标的扰动，将训练数据集PHP与使用CNN提取的特征结合使用，然后采用随机森林回归分别对它们进行分类，然后采用多阶段合奏策略进行最终分类。 事实证明，这种混合方法既准确又高效。在全局级别分类中 ，大多数已发表的工作都集中在基于补丁的分类方法上，以进行全滑坡级别的疾病预测任务。 它可能涉及补丁级别定位以及整个幻灯片级别分类或疾病分级。 这些方法的主要缺点是在整个WSI上执行密集的逐块预测所需的计算时间相对较长。 不同的作品以不同的方式解决了这个问题，有些作品采用启发式采样策略 ，而最近的作品采用任务驱动的基于视觉注意力的粗加工。

Xu et al(2019)²Xu et al(2019)²

Xu et al² in their work adaptively select a sequence of coarse regions from the raw image by a hard visual attention algorithm, and then for each such region, it is able to investigate the abnormal parts based on a soft-attention mechanism. A recurrent network is then built on top to classify the image region and also to predict the location of the image region to be investigated at the next time step. This way, only a fraction of pixels need to be investigated for the classification

Xu et al²在他们的工作中通过硬视觉注意算法自适应地从原始图像中选择了一系列粗糙区域，然后针对每个这样的区域，可以基于软注意机制来研究异常部位。 然后在顶部建立一个递归网络 ，以对图像区域进行分类，并预测下一个时间步骤要研究的图像区域的位置。 这样，只需要调查一小部分像素即可进行分类

Advantages of using visual attention-based models for Whole Slide Image global classification task are:

对整个幻灯片图像全局分类任务使用基于视觉注意的模型的优点是：

• The model tries to learn only the most relevant diagnostically useful areas for disease prediction as it enforces a region selection mechanism.
  该模型尝试执行区域选择机制，因此仅尝试学习对疾病预测最相关的诊断有用区域。
• The model complexity is independent of the size of WSI.
  模型的复杂性与WSI的大小无关。

Another recent work for global classification by Halicek, Martin, et al.³ perform patch-based localization and whole slide classification for Squamous Cell Carcinoma(SCC) and Thyroid Cell Carcinoma using CNN using an entirely different approach. A ground-truth binary mask of the cancer area was produced from each outlined histology slide. The WSIs and corresponding ground-truths were down-sampled by a factor of four using nearest-neighbor interpolation. The downsampled slides were then broken into patches of 101 x 101 size. To ensure generalization the number of image patches was augmented by 8x by applying 90-degree rotations and reflections to develop a more robust diagnostic method. Additionally, to establish a level of color-feature invariance and tolerance to differences in H&E staining between slides, the hue, saturation, brightness, and contrast of each patch were randomly manipulated to make a more rigorous training paradigm before being fed to the Inception-v4 model for detecting head and neck cancer.

Halicek，Martin等人[3]进行的全球分类的另一项最新工作是采用完全不同的方法，使用CNN对鳞状细胞癌(SCC)和甲状腺细胞癌进行基于补丁的定位和整个玻片分类。 从每个概述的组织学幻灯片中得出了癌区域的真相二元掩模。 使用最近邻插值对WSI和相应的地面真相进行4倍下采样。 然后将降采样后的幻灯片分成101 x 101大小的小块。 为了确保通用性，通过应用90度旋转和反射以开发更可靠的诊断方法，图像补丁的数量增加了8倍 。 此外，为了确定幻灯片之间的颜色特征不变性和对H＆E染色差异的容忍度，在将每个补丁的色相，饱和度，亮度和对比度进行随机处理以形成更严格的训练范式后，再将其送入Inception-用于检测头颈癌的v4模型。

监督回归 (Supervised Regression)

In this method, we focus on directly regressing the likelihood of pixel being the center of an object for detection or localization of objects. Regression, unlike classification, gives us a continuous value, usually probability score instead of simply a class label as output. Regression helps in better detection by enforcing topological constraints such as assigning higher probability values to pixels near the object center.Regression also helps with challenges faced in cell/nuclei detection arising due to highly irregular appearance and them occurring as overlapping clumps resulting in problems separating them. Deep regression models proposed in the literature are mainly based on either CNN or Fully Convolutional Network(FCN) architectures.

在这种方法中，我们专注于直接回归以像素为对象中心以检测或定位对象的可能性。 回归与分类不同，回归为我们提供了一个连续的值，通常是概率得分，而不是简单地将类标签作为输出。 回归通过加强拓扑约束(例如为对象中心附近的像素分配更高的概率值)来帮助更好地进行检测;回归还有助于解决由于高度不规则外观而导致的细胞/核检测面临的挑战，并且它们以重叠团块的形式出现，从而导致分离它们的问题。 文献中提出的深度回归模型主要基于CNN或完全卷积网络(FCN)架构。

The paper by Graham et al⁴ on HoVer-Net is one of the most seminal works in the entire area of research. It proposes a unified FCN model for simultaneous nuclear instance segmentation and classification. It leverages the instance-rich information encoded within the vertical and horizontal distances of nuclear pixels to their centers of mass. These distances are then utilized to separate clustered nuclei, resulting in an accurate segmentation, particularly in areas with overlapping instances.Then, for each segmented instance, the network predicts the type of nucleus via a devoted up-sampling branch. The network is composed of three parallel branches that are used for three different tasks. We have corresponding ground truth annotations of the data for each of the three branches.

Graham等人在HoVer-Net上发表的论文是整个研究领域中最具开创性的著作之一。 它为核实例的同时分割和分类提出了一个统一的FCN模型。 它利用了在核像素到其质心的垂直和水平距离内编码的实例丰富的信息。 然后利用这些距离来分离聚集的原子核，从而实现精确的分割，尤其是在实例重叠的区域中。然后，对于每个分割的实例，网络都会通过专用的向上采样分支来预测原子核的类型。 该网络由用于三个不同任务的三个并行分支组成。 对于三个分支中的每个分支，我们都有相应的数据地面真实性注释。

• The Nuclear Pixel(NP) branch predicts whether or not a pixel belongs to the nuclei or background,

  核像素(NP)分支预测像素是否属于原子核或背景，

• whereas the Horizontal-Vertical(HoVer) branch predicts the horizontal and vertical distances of nuclear pixels to their centers of mass. Here the colors represent the gradation of the distance of each nuclear pixel from the center of mass.

  而Horizo​​ntal-Vertical(HoVer)分支预测核像素到其质心的水平和垂直距离。 在这里，颜色表示每个核像素到质心的距离的渐变。

• Blue represents positive distance up to +1 and means the pixel lies on the left side of the COM in case of the horizontal map and above the COM when it comes to the vertical mapping. Similarly, red represents negative distance up to -1 and means the pixel lies on the Right/bottom side of COM accordingly.
  蓝色表示正值，最远为+1，表示像素在水平图的情况下位于COM的左侧，而在垂直图的情况下位于COM的上方。 同样，红色表示负距离，最大为-1，表示像素相应地位于COM的右/下侧。

• Then, the Nuclear Classification(NC) branch(optional) predicts the type of nucleus for each pixel.

  然后， 核分类(NC)分支(可选)预测每个像素的核类型。

Graham et al(2019)⁴Graham等(2019)⁴

In particular, the NP and HoVer branches jointly achieve nuclear instance segmentation by first separating nuclear pixels from the background (NP branch) and then separating touching nuclei (HoVer branch). This is the same model that was used for the localization and clustering of tissues step for modeling Whole Slide Images as graphs for subsequent learning using Graph Neural Networks, as discussed in the previous post.

特别是， NP和HoVer分支首先通过将核像素与背景分离(NP分支)，然后分离接触核( HoVer分支 )，共同实现核实例分割。 这是用于组织的定位和聚类步骤的模型，该步骤用于将整个幻灯片图像建模为图形，以供以后使用Graph Neural Networks学习，如先前的文章所述。

In the next and final part of our discussion on Deep Learning in Histopathology, we shall be discussing Supervised Segmentation, Weakly Supervised and Unsupervised Learning methodologies in the context of Digital Histopathology, with appropriate application and relevant literature.

在关于组织病理学深度学习的讨论的下一部分和最后一部分中，我们将讨论数字组织病理学背景下的监督分割，弱监督和无监督学习方法，并结合适当的应用和相关文献。

PS: I have linked technically important terms to respective resources explaining them.

PS：我已将技术上重要的术语链接到解释它们的相应资源。

翻译自: https://towardsdatascience.com/deep-learning-in-histopathology-35c0294d38eb

基于深度学习的病理