1. Kernel代码copy至GPU设备上
When using the CUDA runtime API, kernel code is downloaded to the device once. This happens as an implicit action right after CUDA runtime context creation. When using the CUDA driver API, the app has control over when kernels get downloaded, and how often. It seems this is not currently covered by the CUDA documentation, I will file an enhancement request for that.
2. JIT(Just In Time compilation)
PTX通用代码-->SASS设备相关代码
有可能在运行时进行编译
3. 缓存加载等操作
4. Context初始化
5. 打开cuDNN的auto tuner,一上来先尝试多种不同的kernel,后面就用尝试的几种kernel里最快的那个来跑。"torch.backends.cudnn.benchmark=True"