Reproduce

1. Pytorch关于不确定性的描述文档

2. 主要代码

   import torch
   import random
   import numpy
   
   
   def init_seeds(seed=20):
       print('seed :', seed)
       torch.manual_seed(seed) #为所有CPU和GPU设置随机数种子
       torch.cuda.manual_seed(seed) #为GPU cuda:0设置随机数种子
       torch.cuda.manual_seed_all(seed) #为所有GPU设置随机数种子
       random.seed(seed) #设置python随机数种子
       np.random.seed(seed) #设置numpy随机数种子
       torch.backends.cudnn.benchmark = False #关闭卷积算法的自动选优
       torch.backends.cudnn.deterministic = True #设置算法本身的确定性
       torch.use_deterministic_algorithms(True) #设置pytorch使用确定的算法而不是不确定的算法，某些算法不支持确定的算法会报错
       
   

3. 数据加载器

   DataLoader will reseed workers following Randomness in multi-process data loading algorithm. Use worker_init_fn() and generator to preserve reproducibility:

   def seed_worker(worker_id):
       worker_seed = torch.initial_seed() % 2**32
       numpy.random.seed(worker_seed)
       random.seed(worker_seed)
   
   g = torch.Generator()
   g.manual_seed(20) #注意此处种子的一致性
   
   DataLoader(
       train_dataset,
       batch_size=batch_size,
       num_workers=num_workers,
       worker_init_fn=seed_worker,
       generator=g,
   )
   

4. 此外，可能需要设置环境变量：

   • provide a separate workspace for each used stream using the cublasSetWorkspace() function, or
   • have one cuBLAS handle per stream, or
   • use cublasLtMatmul() instead of gemm() family of functions and provide user owned workspace, or
   • set a debug environment variable CUBLAS_WORKSPACE_CONFIG to “:16:8” (may limit overall performance) or “:4096:8” (will increase library footprint in GPU memory by approximately 24MiB).

   Any of those settings will allow for deterministic behavior even with multiple concurrent streams sharing a single cuBLAS handle.

   一般情况下设置CUBLAS_WORKSPACE_CONFIG=:4096:8即可

经过实验，torch.use_deterministic_algorithms(True)极大影响YOLOv3算法的性能，mAP基本减半，但不设置该项则无法保证可复现性。