nnUNet

nnunet的readme文档

[2020_10_21] Update: We now have documentation for common questions and
common issues. We now also provide reference epoch times for
several datasets and tips on how to identify bottlenecks.

Please read these documents before opening a new issue!

[2020_10_21] 更新: 现在，我们拥有有关常见问题和常见问题的文档，我们现在还提供了一些数据集的参考纪元时间，以及有关如何识别瓶颈的提示。

在打开新期刊之前，请阅读这些文档！

nnU-Net

In 3D biomedical image segmentation, dataset properties like imaging modality, image sizes, voxel spacings, class
ratios etc vary drastically.
For example, images in the Liver and Liver Tumor Segmentation Challenge dataset
are computed tomography (CT) scans, about 512x512x512 voxels large, have isotropic voxel spacings and their
intensity values are quantitative (Hounsfield Units).
The Automated Cardiac Diagnosis Challenge dataset on the other hand shows cardiac
structures in cine MRI with a typical image shape of 10x320x320 voxels, highly anisotropic voxel spacings and
qualitative intensity values. In addition, the ACDC dataset suffers from slice misalignments and a heterogeneity of
out-of-plane spacings which can cause severe interpolation artifacts if not handled properly.

在3D生物医学图像分割中，数据集属性（如成像方式，图像大小，体素间距，类别)比率等变化很大。
例如，肝脏和肝脏肿瘤分割挑战数据集中的图像
是计算机断层扫描（CT)扫描，大约512x512x512大体素，各向同性体素间距及其强度值是定量的（Hounsfield单位)。

另一方面，自动心脏诊断挑战数据集显示了心脏
电影MRI中的结构，典型图像形状为10x320x320体素，高度各向异性的体素间距和
定性强度值。 此外，ACDC数据集还存在切片未对齐和异质性的问题。
平面外的间距，如果处理不当，可能会导致严重的插值伪像。

In current research practice, segmentation pipelines are designed manually and with one specific dataset in mind.
Hereby, many pipeline settings depend directly or indirectly on the properties of the dataset
and display a complex co-dependence: image size, for example, affects the patch size, which in
turn affects the required receptive field of the network, a factor that itself influences several other
hyperparameters in the pipeline. As a result, pipelines that were developed on one (type of) dataset are inherently
incomaptible with other datasets in the domain.

在当前的研究实践中，分段管道是手动设计的，并且要考虑一个特定的数据集。
因此，许多管道设置直接或间接取决于数据集的属性
并显示出复杂的相互依赖性：例如，图像大小会影响patch大小，
转向会影响网络所需的接收域，这个因素本身会影响其他几个
管道中的超参数。 结果，在一个（类型)数据集上开发的管道本质上是
与域中的其他数据集不兼容。

nnU-Net is the first segmentation method that is designed to deal with the dataset diversity found in the domain. It
condenses and automates the keys decisions for designing a successful segmentation pipeline for any given dataset.

nnU-Net是第一种旨在处理在域中发现的数据集多样性的分割方法。 它
为任何给定的数据集设计成功的分割管道的关键决策，都将得到浓缩和自动化。

nnU-Net makes the following contributions to the field:

1. Standardized baseline: nnU-Net is the first standardized deep learning benchmark in biomedical segmentation.
   Without manual effort, researchers can compare their algorithms against nnU-Net on an arbitrary number of datasets
   to provide meaningful evidence for proposed improvements.
2. Out-of-the-box segmentation method: nnU-Net is the first plug-and-play tool for state-of-the-art biomedical
   segmentation. Inexperienced users can use nnU-Net out of the box for their custom 3D segmentation problem without
   need for manual intervention.
3. Framework: nnU-Net is a framework for fast and effective development of segmentation methods. Due to its modular
   structure, new architectures and methods can easily be integrated into nnU-Net. Researchers can then benefit from its
   generic nature to roll out and evaluate their modifications on an arbitrary number of datasets in a
   standardized environment.

nnU-Net在该领域做出了以下贡献：

1. 标准化基准： nnU-Net是生物医学细分领域的第一个标准化深度学习基准。
   研究人员无需人工即可在任意数量的数据集上将其算法与nnU-Net进行比较
   为拟议的改进提供有意义的证据。
2. 开箱即用的分割方法： nnU-Net是第一个用于最新生物医学分割的即插即用工具。 没有经验的用户可以直接使用nnU-Net解决其自定义3D分割问题，而无需手动干预。
3. 框架： nnU-Net是用于快速有效开发细分方法的框架。 由于其模块化的结构，可以轻松将新的体系结构和方法集成到nnU-Net中。 然后，研究人员可以从其通用性质中受益，以在标准化环境中对任意数量的数据集进行发布和评估其修改。

For more information about nnU-Net, please read the following paper:

Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2020). nnU-Net: a self-configuring method 
for deep learning-based biomedical image segmentation. Nature Methods, 1-9.

Please also cite this paper if you are using nnU-Net for your research!

Table of Contents

• nnU-Net
• Table of Contents
• 目录
• Installation
• 安装
• Usage
• 用法
  • How to run nnU-Net on a new dataset
  • 如何在新数据集上运行nnU-Net
    • Dataset conversion
    • 数据集转换
    • Experiment planning and preprocessing
    • Model training
      • 2D U-Net
      • 3D full resolution U-Net
      • 3D U-Net cascade
        • 3D low resolution U-Net
        • 3D full resolution U-Net
      • Multi GPU training
    • Identifying the best U-Net configuration
    • Run inference
  • How to run inference with pretrained models
  • Examples
• Extending or Changing nnU-Net
• Information on run time and potential performance bottlenecks.
• Common questions and issues

目录

• nnU-Net
• Table of Contents
• 目录
• Installation
• 安装
• Usage
• 用法
  • How to run nnU-Net on a new dataset
  • 如何在新数据集上运行nnU-Net
    • Dataset conversion
    • 数据集转换
    • Experiment planning and preprocessing
    • Model training
      • 2D U-Net
      • 3D full resolution U-Net
      • 3D U-Net cascade
        • 3D low resolution U-Net
        • 3D full resolution U-Net
      • Multi GPU training
    • Identifying the best U-Net configuration
    • Run inference
  • How to run inference with pretrained models
  • Examples
• Extending or Changing nnU-Net
• Information on run time and potential performance bottlenecks.
• Common questions and issues

Installation

nnU-Net has been tested on Linux (Ubuntu 16, 18 and 20; centOS, RHEL). We do not provide support for other operating
systems.

nnU-Net requires a GPU! For inference, the GPU should have 4 GB of VRAM. For training nnU-Net models the GPU should have at
least 10 GB (popular non-datacenter options are the RTX 2080ti, RTX 3080 or RTX 3090). Due to the use of automated mixed
precision, fastest training times are achieved with the Volta architecture (Titan V, V100 GPUs) when installing pytorch
the easy way. Since pytorch comes with cuDNN 7.6.5 and tensor core acceleration on Turing GPUs is not supported for 3D
convolutions in this version, you will not get the best training speeds on Turing GPUs. You can remedy that by compiling pytorch from source
(see here) using cuDNN 8.0.2 or newer. This will unlock Turing GPUs
(RTX 2080ti, RTX 6000) for automated mixed precision training with 3D convolutions and make the training blistering
fast as well. Note that future versions of pytorch may include cuDNN 8.0.2 or newer by default and
compiling from source will not be necessary.
We don’t know the speed of Ampere GPUs with vanilla vs self-compiled pytorch yet - this section will be updated as
soon as we know.

安装

nnU-Net已在Linux（Ubuntu 16、18和20； centOS，RHEL）上进行了测试。我们不为其他操作系统提供支持。

nnU-Net需要GPU！为了进行推断，GPU应该具有4 GB的VRAM。为了训练nnU-Net模型，GPU应该至少具有10 GB（流行的非数据中心选项是RTX 2080ti，RTX 3080或RTX 3090）。由于使用自动混合
简易安装pytorch时，使用Volta架构（Titan V，V100 GPU）可实现精确，最快的培训时间。由于pytorch随附cuDNN 7.6.5，并且此版本的3D卷积不支持Turing GPU上的张量核心加速，因此您将无法在Turing GPU上获得最佳的训练速度。您可以通过使用cuDNN 8.0.2或更高版本从源代码编译pytorch(请参阅here)来解决这一问题。这将解锁Turing GPU（RTX 2080ti，RTX 6000），以进行3D卷积的自动混合精度训练，并使训练起泡速度也更快。请注意，默认情况下pytorch的未来版本可能包括cuDNN 8.0.2或更高版本，因此无需从源代码进行编译。
我们尚不知道带有香草vs自编译pytorch的Ampere GPU的速度-我们将尽快更新此部分。

For training, we recommend a strong CPU to go along with the GPU. At least 6 CPU cores (12 threads) are recommended. CPU
requirements are mostly related to data augmentation and scale with the number of input channels. They are thus higher
for datasets like BraTS which use 4 image modalities and lower for datasets like LiTS which only uses CT images.

为了进行培训，我们建议将强大的CPU与GPU搭配使用。 建议至少使用6个CPU内核（12个线程）。 CPU需求主要与数据扩充和输入通道数量的扩展有关。 因此，对于使用4种图像模态的BraTS等数据集，其较高；对于仅使用CT图像的LiTS等数据集，其较低。

We very strongly recommend you install nnU-Net in a virtual environment.
Here is a quick how-to for Ubuntu.
If you choose to compile pytorch from source, you will need to use conda instead of pip. In that case, please set the environment variable OMP_NUM_THREADS=1 (preferably in your bashrc using export OMP_NUM_THREADS=1). This is important!

我们强烈建议您在虚拟环境中安装nnU-Net。
这是Ubuntu的快速使用方法。
如果选择从源代码编译pytorch，则需要使用conda而不是pip。 在这种情况下，请设置环境变量OMP_NUM_THREADS = 1（最好在您的bashrc中使用" export OMP_NUM_THREADS = 1"）。 这个很重要！

Python 2 is deprecated and not supported. Please make sure you are using Python 3.

1. Install PyTorch. You need at least version 1.6
2. Install nnU-Net depending on your use case:

   1. For use as standardized baseline, out-of-the-box segmentation algorithm or for running inference with pretrained models:

   2. pip install nnunet

   3. For use as integrative framework (this will create a copy of the nnU-Net code on your computer so that you can modify it as needed):

      git clone https://github.com/MIC-DKFZ/nnUNet.git
      cd nnUNet
      pip install -e .
      

3. nnU-Net needs to know where you intend to save raw data, preprocessed data and trained models. For this you need to
   set a few of environment variables. Please follow the instructions here.
4. (OPTIONAL) Install hiddenlayer. hiddenlayer enables nnU-net to generate
   plots of the network topologies it generates (see Model training). To install hiddenlayer,

不推荐使用Python 2，不支持该功能。请确保您使用的是Python 3。

1)安装PyTorch。您至少需要1.6版
2)根据您的使用情况安装nnU-Net：

1)可用作**标准化基线**，即用型**分割算法**或用于**使用预训练模型进行推论**:
  
     ```pip install nnunet```

2)用作集成**框架（这将在您的计算机上创建nnU-Net代码的副本，以便您可以根据需要对其进行修改)：

      ```bash
      git clone https://github.com/MIC-DKFZ/nnUNet.git
      cd nnUNet
      pip install -e .
      ```

3)nnU-Net需要知道您打算将原始数据，预处理数据和训练有素的模型保存在何处。为此，您需要
设置一些环境变量。请按照此处的说明进行操作。
4)（可选)安装hiddenlayer。 hiddenlayer使nnU-net能够生成
生成的网络拓扑图(请参见模型训练)。要安装hiddenlayer，
运行以下命令：

```bash
pip install --upgrade git+https://github.com/nanohanno/hiddenlayer.git@bugfix/get_trace_graph#egg=hiddenlayer
```

Installing nnU-Net will add several new commands to your terminal. These commands are used to run the entire nnU-Net
pipeline. You can execute them from any location on your system. All nnU-Net commands have the prefix nnUNet_ for
easy identification.

安装nnU-Net将向您的终端添加几个新命令。 这些命令用于运行整个nnU-Net管道。 您可以从系统上的任何位置执行它们。 所有nnU-Net命令都带有前缀nnUNet_，以便于识别。

Note that these commands simply execute python scripts. If you installed nnU-Net in a virtual environment, this
environment must be activated when executing the commands.

请注意，这些命令仅执行python脚本。 如果在虚拟环境中安装了nnU-Net，则在执行命令时必须激活该环境。

All nnU-Net commands have a -h option which gives information on how to use them.

A typical installation of nnU-Net can be completed in less than 5 minutes. If pytorch needs to be compiled from source
(which is what we currently recommend when using Turing GPUs), this can extend to more than an hour.

安装nnU-Net将向您的终端添加几个新命令。 这些命令用于运行整个nnU-Net
管道。 您可以从系统上的任何位置执行它们。 所有nnU-Net命令的前缀均为nnUNet_
容易识别。

请注意，这些命令仅执行python脚本。 如果您在虚拟环境中安装了nnU-Net，则此
执行命令时必须激活环境。

所有nnU-Net命令都有一个-h选项，该选项提供有关如何使用它们的信息。

nnU-Net的典型安装可以在不到5分钟的时间内完成。 如果pytorch需要从源代码编译
（这是我们目前在使用Turing GPU时的建议），这可能会延长一个多小时。

Usage

To familiarize yourself with nnU-Net we recommend you have a look at the Examples before you start with
your own dataset.

用法

为了使您熟悉nnU-Net，我们建议您先阅读Examples
您自己的数据集。

How to run nnU-Net on a new dataset

Given some dataset, nnU-Net fully automatically configures an entire segmentation pipeline that matches its properties.
nnU-Net covers the entire pipeline, from preprocessing to model configuration, model training, postprocessing
all the way to ensembling. After running nnU-Net, the trained model(s) can be applied to the test cases for inference.

如何在新数据集上运行nnU-Net

给定一些数据集，nnU-Net会完全自动配置与其属性匹配的整个分段管道。
nnU-Net涵盖了从预处理到模型配置，模型训练，后处理的整个流程
一直到合奏。 在运行nnU-Net之后，可以将训练后的模型应用于测试用例以进行推断。

Dataset conversion

nnU-Net expects datasets in a structured format. This format closely (but not entirely) follows the data structure of
the Medical Segmentation Decthlon. Please read
this for information on how to convert datasets to be compatible with nnU-Net.

Experiment planning and preprocessing

As a first step, nnU-Net extracts a dataset fingerprint (a set of dataset-specific properties such as
image sizes, voxel spacings, intensity information etc). This information is used to create three U-Net configurations:
a 2D U-Net, a 3D U-Net that operated on full resolution images as well as a 3D U-Net cascade where the first U-Net
creates a coarse segmentation map in downsampled images which is then refined by the second U-Net.

Provided that the requested raw dataset is located in the correct folder (nnUNet_raw_data_base/nnUNet_raw_data/TaskXXX_MYTASK,
also see here), you can run this step with the following command:

nnUNet_plan_and_preprocess -t XXX --verify_dataset_integrity

XXX is the integer identifier associated with your Task name TaskXXX_MYTASK. You can pass several task IDs at once.

Running nnUNet_plan_and_preprocess will populate your folder with preprocessed data. You will find the output in
nnUNet_preprocessed/TaskXXX_MYTASK. nnUNet_plan_and_preprocess creates subfolders with preprocessed data for the 2D
U-Net as well as all applicable 3D U-Nets. It will also create ‘plans’ files (with the ending.pkl) for the 2D and
3D configurations. These files contain the generated segmentation pipeline configuration and will be read by the
nnUNetTrainer (see below). Note that the preprocessed data folder only contains the training cases.
The test images are not preprocessed (they are not looked at at all!). Their preprocessing happens on the fly during
inference.

运行nnUNet_plan_and_preprocess 将会构架预处理好的数据。你可以在nnUnet_preprocessed/TaskXXX_MYTASK里面找到输出。nnUNet_plan_and_preprocess 创建了2DUnet和3DUnet预处理数据的子文件夹。它任然创建’plan’文件（包括ending.pkl）对于2D和3D的配置。 这些文件包含生成分段基线配置，可以被nnUnetTrainer读取（您接着往下看）。记住预处理文件夹只包含训练样例。测试图片根本不会被预处理，看都不看。他们的预处理是在飞行通道中发生的。

--verify_dataset_integrity should be run at least for the first time the command is run on a given dataset. This will execute some
checks on the dataset to ensure that it is compatible with nnU-Net. If this check has passed once, it can be
omitted in future runs. If you adhere to the dataset conversion guide (see above) then this should pass without issues 😃

--verify_dataset_integrity 至少在第一次运行给定数据的时候需要运行。他会执行来保障数据集可以被nnUnet执行。如果检查通过了就不需要继续了。如果你坚持采用我们提供的数据转换就应该没问题。

Note that nnUNet_plan_and_preprocess accepts several additional input arguments. Running -h will list all of them
along with a description. If you run out of RAM during preprocessing, you may want to adapt the number of processes
used with the -tl and -tf options.

注意：nnUNet_plan_and_preprocess 接受几个另外的输入参数。运行-h 可以列出来所有的描述。如果你在预处理的时候用光了RAM，你就要调整进程的数量，使用-tl 和-tf 操作。

After nnUNet_plan_and_preprocess is completed, the U-Net configurations have been created and a preprocessed copy
of the data will be located at nnUNet_preprocessed/TaskXXX_MYTASK.

在nnUNet_plan_and_preprocess 完成了之后，U-Net配置被创建和预处理的文件被拷贝到nnUNet_preprocessed/TaskXXX_MYTASK。

Extraction of the dataset fingerprint can take from a couple of seconds to several minutes depending on the properties
of the segmentation task. Pipeline configuration given the extracted finger print is nearly instantaneous (couple
of seconds). Preprocessing depends on image size and how powerful the CPU is. It can take between seconds and several
tens of minutes.

提取数据指纹可能需要一定的时间，这取决于分割任务数据集的种类。基线配置几乎是瞬间完成的。预处理取决于图片的大小和CPU的性能如何，时间从几秒到十几分钟不同。

Model training

nnU-Net trains all U-Net configurations in a 5-fold cross-validation. This enables nnU-Net to determine the
postprocessing and ensembling (see next step) on the training dataset. Per default, all U-Net configurations need to
be run on a given dataset. There are, however situations in which only some configurations (and maybe even without
running the cross-validation) are desired. See FAQ for more information.

模型训练

nnU-Net 训练 所有的 U-Net配置 在五折交叉验证上。这使得nnU-Net可以在训练数据集上进行后处理和集成。默认情况下，所有的U-Net配置都需要运行在给定的数据集中，在某些情况下，只有一部分配置是需要的。可以看FAQ看到更多的内容。

Note that not all U-Net configurations are created for all datasets. In datasets with small image sizes, the U-Net
cascade is omitted because the patch size of the full resolution U-Net already covers a large part of the input images.

注意：不是所有的U-Net配置被创建在所有的数据集上。对于小的图片数据集，U-Net级联被忽略了因为patch size对于全解决方案的U-Net已经覆盖了大范围的输入图片。

Training models is done with the nnUNet_train command. The general structure of the command is:

nnUNet_train CONFIGURATION TRAINER_CLASS_NAME TASK_NAME_OR_ID FOLD  --npz (additional options)

训练模型使用nnUNet_train命令。一般架构的命令是：

nnUNet_train Configuration Trainer_class_name Task_name_or_id Fold --npz (additional options)

CONFIGURATION is a string that identifies the requested U-Net configuration. TRAINER_CLASS_NAME is the name of the
model trainer. If you implement custom trainers (nnU-Net as a framework) you can specify your custom trainer here.
TASK_NAME_OR_ID specifies what dataset should be trained on and FOLD specifies which fold of the 5-fold-cross-validaton
is trained.

Configuration 是标志请求U-Net配置的字符串。Train_class_name 是训练模型的名称。如果您实现了自定义训练器(nnU-Net作为一个框架)，您可以在这里指定您的自定义训练器。TASK_NAME_OR_ID指定应该训练的数据集，FOLD指定5- FOLD交叉验证的折叠。

nnU-Net stores a checkpoint every 50 epochs. If you need to continue a previous training, just add a -c to the
training command.

nnU-Net 每50个epoch 存储一个checkpoint 。 如果你需要继续一个先前的训练，可以使用-C训练指令。

IMPORTANT: --npz makes the models save the softmax outputs during the final validation. It should only be used for trainings
where you plan to run nnUNet_find_best_configuration afterwards
(this is nnU-Nets automated selection of the best performing (ensemble of) configuration(s), see below). If you are developing new
trainer classes you may not need the softmax predictions and should therefore omit the --npz flag. Exported softmax
predictions are very large and therefore can take up a lot of disk space.

重要：--npz 使得模型最后保存输出一个softmax在最后的预测上。所以只能用在训练上当你之后打算运行nnUNet_find_best_configuration的时候。nnUnet自动选择最好的表现和配置。可以从下面看到。如果你正在开发新的
你可能不需要softmax预测，因此应该省略’ -npz '标志。softmax预测导出非常大，因此会占用大量磁盘空间。

If you ran initially without the --npz flag but now require the softmax predictions, simply run

nnUNet_train CONFIGURATION TRAINER_CLASS_NAME TASK_NAME_OR_ID FOLD -val --npz

to generate them. This will only rerun the validation, not the training.

如果你使用soft Max进行预测但是你没有使用—npz 标志可以使用xxxxx去生成他们，这只能返回验证，不会返回训练。

See nnUNet_train -h for additional options.

2D U-Net

For FOLD in [0, 1, 2, 3, 4], run:

nnUNet_train 2d nnUNetTrainerV2 TaskXXX_MYTASK FOLD --npz

3D full resolution U-Net

For FOLD in [0, 1, 2, 3, 4], run:

nnUNet_train 3d_fullres nnUNetTrainerV2 TaskXXX_MYTASK FOLD --npz

3D U-Net cascade

3D low resolution U-Net

For FOLD in [0, 1, 2, 3, 4], run:

nnUNet_train 3d_lowres nnUNetTrainerV2 TaskXXX_MYTASK FOLD --npz

3D full resolution U-Net

For FOLD in [0, 1, 2, 3, 4], run:

nnUNet_train 3d_cascade_fullres nnUNetTrainerV2CascadeFullRes TaskXXX_MYTASK FOLD --npz

Note that the 3D full resolution U-Net of the cascade requires the five folds of the low resolution U-Net to be
completed beforehand!

注意：3d全分辨率集联unet需要五折交叉验证，低分辨率unet被完成之前。

The trained models will we written to the RESULTS_FOLDER/nnUNet folder. Each training obtains an automatically generated

训练模型会被写入结果文件夹/nnunet，每一个训练都是被自动生成的。

output folder name:

nnUNet_preprocessed/CONFIGURATION/TaskXXX_MYTASKNAME/TRAINER_CLASS_NAME__PLANS_FILE_NAME/FOLD

For Task002_Heart (from the MSD), for example, this looks like this:

RESULTS_FOLDER/nnUNet/
├── 2d
│   └── Task02_Heart
│       └── nnUNetTrainerV2__nnUNetPlansv2.1
│           ├── fold_0
│           ├── fold_1
│           ├── fold_2
│           ├── fold_3
│           └── fold_4
├── 3d_cascade_fullres
├── 3d_fullres
│   └── Task02_Heart
│       └── nnUNetTrainerV2__nnUNetPlansv2.1
│           ├── fold_0
│           │   ├── debug.json
│           │   ├── model_best.model
│           │   ├── model_best.model.pkl
│           │   ├── model_final_checkpoint.model
│           │   ├── model_final_checkpoint.model.pkl
│           │   ├── network_architecture.pdf
│           │   ├── progress.png
│           │   └── validation_raw
│           │       ├── la_007.nii.gz
│           │       ├── la_007.pkl
│           │       ├── la_016.nii.gz
│           │       ├── la_016.pkl
│           │       ├── la_021.nii.gz
│           │       ├── la_021.pkl
│           │       ├── la_024.nii.gz
│           │       ├── la_024.pkl
│           │       ├── summary.json
│           │       └── validation_args.json
│           ├── fold_1
│           ├── fold_2
│           ├── fold_3
│           └── fold_4
└── 3d_lowres

Note that 3d_lowres and 3d_cascade_fullres are not populated because this dataset did not trigger the cascade. In each
model training output folder (each of the fold_x folder, 10 in total here), the following files will be created (only
shown for one folder above for brevity):

• debug.json: Contains a summary of blueprint and inferred parameters used for training this model. Not easy to read,
  but very useful for debugging 😉
• model_best.model / model_best.model.pkl: checkpoint files of the best model identified during training. Not used right now.
• model_final_checkpoint.model / model_final_checkpoint.model.pkl: checkpoint files of the final model (after training
  has ended). This is what is used for both validation and inference.
• network_architecture.pdf (only if hiddenlayer is installed!): a pdf document with a figure of the network architecture in it.
• progress.png: A plot of the training (blue) and validation (red) loss during training. Also shows an approximation of
  the evlauation metric (green). This approximation is the average Dice score of the foreground classes. It should,
  however, only to be taken with a grain of salt because it is computed on randomly drawn patches from the validation
  data at the end of each epoch, and the aggregation of TP, FP and FN for the Dice computation treats the patches as if
  they all originate from the same volume (‘global Dice’; we do not compute a Dice for each validation case and then
  average over all cases but pretend that there is only one validation case from which we sample patches). The reason for
  this is that the ‘global Dice’ is easy to compute during training and is still quite useful to evaluate whether a model
  is training at all or not. A proper validation is run at the end of the training.
• validation_raw: in this folder are the predicted validation cases after the training has finished. The summary.json
  contains the validation metrics (a mean over all cases is provided at the end of the file).

During training it is often useful to watch the progress. We therefore recommend that you have a look at the generated
progress.png when running the first training. It will be updated after each epoch.

Training times largely depend on the GPU. The smallest GPU we recommend for training is the Nvidia RTX 2080ti. With
this GPU (and pytorch compiled with cuDNN 8.0.2), all network trainings take less than 2 days.

Multi GPU training

Multi GPU training is experimental and NOT RECOMMENDED!

nnU-Net supports two different multi-GPU implementation: DataParallel (DP) and Distributed Data Parallel (DDP)
(but currently only on one host!). DDP is faster than DP and should be preferred if possible. However, if you did not
install nnunet as a framework (meaning you used the pip install nnunet variant), DDP is not available. It requires a
different way of calling the correct python script (see below) which we cannot support from our terminal commands.

Distributed training currently only works for the basic trainers (2D, 3D full resolution and 3D low resolution) and not
for the second, high resolution U-Net of the cascade. The reason for this is that distributed training requires some
changes to the network and loss function, requiring a new nnUNet trainer class. This is, as of now, simply not
implemented for the cascade, but may be added in the future.

To run distributed training (DP), use the following command:

CUDA_VISIBLE_DEVICES=0,1,2... nnUNet_train_DP CONFIGURATION nnUNetTrainerV2_DP TASK_NAME_OR_ID FOLD -gpus GPUS --dbs

Note that nnUNetTrainerV2 was replaced with nnUNetTrainerV2_DP. Just like before, CONFIGURATION can be 2d, 3d_lowres or
3d_fullres. TASK_NAME_OR_ID refers to the task you would like to train and FOLD is the fold of the cross-validation.
GPUS (integer value) specifies the number of GPUs you wish to train on. To specify which GPUs you want to use, please make use of the
CUDA_VISIBLE_DEVICES envorinment variable to specify the GPU ids (specify as many as you configure with -gpus GPUS).
–dbs, if set, will distribute the batch size across GPUs. So if nnUNet configures a batch size of 2 and you run on 2 GPUs
, each GPU will run with a batch size of 1. If you omit --dbs, each GPU will run with the full batch size (2 for each GPU
in this example for a total of batch size 4).

To run the DDP training you must have nnU-Net installed as a framework. Your current working directory must be the
nnunet folder (the one that has the dataset_conversion, evaluation, experiment_planning, … subfolders!). You can then run
the DDP training with the following command:

CUDA_VISIBLE_DEVICES=0,1,2... python -m torch.distributed.launch --master_port=XXXX --nproc_per_node=Y run/run_training_DDP.py CONFIGURATION nnUNetTrainerV2_DDP TASK_NAME_OR_ID FOLD --dbs

XXXX must be an open port for process-process communication (something like 4321 will do on most systems). Y is the
number of GPUs you wish to use. Remember that we do not (yet) support distributed training across compute nodes. This
all happens on the same system. Again, you can use CUDA_VISIBLE_DEVICES=0,1,2 to control what GPUs are used.
If you run more than one DDP training on the same system (say you have 4 GPUs and you run two training with 2 GPUs each)
you need to specify a different --master_port for each training!

IMPORTANT!
Multi-GPU training results in models that cannot be used for inference easily (as said above, all of this is experimental 😉 ).
After finishing the training of all folds, run nnUNet_change_trainer_class on the folder where the trained model is
(see nnUNet_change_trainer_class -h for instructions). After that you can run inference.

Identifying the best U-Net configuration

Once all models are trained, use the following
command to automatically determine what U-Net configuration(s) to use for test set prediction:

nnUNet_find_best_configuration -m 2d 3d_fullres 3d_lowres 3d_cascade_fullres -t XXX --strict

(all 5 folds need to be completed for all specified configurations!)

On datasets for which the cascade was not configured, use -m 2d 3d_fullres instead. If you wish to only explore some
subset of the configurations, you can specify that with the -m command. We recommend setting the
--strict (crash if one of the requested configurations is
missing) flag. Additional options are available (use -h for help).

Run inference

Remember that the data located in the input folder must adhere to the format specified
here.

nnUNet_find_best_configuration will print a string to the terminal with the inference commands you need to use.
The easiest way to run inference is to simply use these commands.

If you wish to manually specify the configuration(s) used for inference, use the following commands:

For each of the desired configurations, run:

nnUNet_predict -i INPUT_FOLDER -o OUTPUT_FOLDER -t TASK_NAME_OR_ID -m CONFIGURATION --save_npz

Only specify --save_npz if you intend to use ensembling. --save_npz will make the command save the softmax
probabilities alongside of the predicted segmentation masks requiring a lot of disk space.

Please select a separate OUTPUT_FOLDER for each configuration!

If you wish to run ensembling, you can ensemble the predictions from several configurations with the following command:

nnUNet_ensemble -f FOLDER1 FOLDER2 ... -o OUTPUT_FOLDER -pp POSTPROCESSING_FILE

You can specify an arbitrary number of folders, but remember that each folder needs to contain npz files that were
generated by nnUNet_predict. For ensembling you can also specify a file that tells the command how to postprocess.
These files are created when running nnUNet_find_best_configuration and are located in the respective trained model
directory (RESULTS_FOLDER/nnUNet/CONFIGURATION/TaskXXX_MYTASK/TRAINER_CLASS_NAME__PLANS_FILE_IDENTIFIER/postprocessing.json or
RESULTS_FOLDER/nnUNet/ensembles/TaskXXX_MYTASK/ensemble_X__Y__Z–X__Y__Z/postprocessing.json). You can also choose to
not provide a file (simply omit -pp) and nnU-Net will not run postprocessing.

Note that per default, inference will be done with all available folds. We very strongly recommend you use all 5 folds.
Thus, all 5 folds must have been trained prior to running inference. The list of available folds nnU-Net found will be
printed at the start of the inference.

How to run inference with pretrained models

Trained models for all challenges we participated in are publicly available. They can be downloaded and installed
directly with nnU-Net. Note that downloading a pretrained model will overwrite other models that were trained with
exactly the same configuration (2d, 3d_fullres, …), trainer (nnUNetTrainerV2) and plans.

To obtain a list of available models, as well as a short description, run

nnUNet_print_available_pretrained_models

You can then download models by specifying their task name. For the Liver and Liver Tumor Segmentation Challenge,
for example, this would be:

nnUNet_download_pretrained_model Task029_LiTS

After downloading is complete, you can use this model to run inference. Keep in mind that each of
these models has specific data requirements (Task029_LiTS runs on abdominal CT scans, others require several image
modalities as input in a specific order).

When using the pretrained models you must adhere to the license of the dataset they are trained on! If you run
nnUNet_download_pretrained_model you will find a link where you can find the license for each dataset.

Examples

To get you started we compiled two simple to follow examples:

• run a training with the 3d full resolution U-Net on the Hippocampus dataset. See here.
• run inference with nnU-Net’s pretrained models on the Prostate dataset. See here.

Usability not good enough? Let us know!

Extending or Changing nnU-Net

Please refer to this guide.

Information on run time and potential performance bottlenecks.

We have compiled a list of expected epoch times on standardized datasets across many different GPUs. You can use them
to verify that your system is performing as expected. There are also tips on how to identify bottlenecks and what
to do about them.

Click here.

Common questions and issues

We have collected solutions to common questions and
problems. Please consult these documents before you open a new issue.