django部署pytorch框架，显存溢出后，服务持续运行，却无法上传图片解决方案

对于Django部署的项目，基于pytorch框架的metaSR，产生了显存溢出后，后续服务仍在运行，但是diango无法上传图片，如下：

需要重新启动服务，不符合上线要求。

RuntimeError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 0; 9.92 GiB total capacity; 4.51 GiB already allocated; 1.30 GiB free; 391.40 MiB cached)
    
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>

修改代码如下:

try:

      ...........

except RuntimeError as e:
      if "out of memory" in str(e):
             print("WARNING: out of memory")
             if hasattr(torch.cuda, 'empty_cache'):
                    torch.cuda.empty_cache()
      else:
             raise exception
      self.output.put((uid, constants.IMG_FAILED, "The size of picture is too large, cache is out of memory"))
      continue

对于MetaSR，运行完要释放cache，程序后加torch.cuda.empty_cache().

^Croot@localhost:/home/AI/services/app/CV/Django_MetaRDN# python manage.py runserver 0.0.0.0:9000
Performing system checks...
 
Loading model from meta_SR/model_1000.pt
load_model_mode=1
System check identified no issues (0 silenced).
August 10, 2019 - 03:07:38
Django version 2.1, using settings 'Django_MetaRDN.settings'
Starting development server at http://0.0.0.0:9000/
Quit the server with CONTROL-C.
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
Input scale is  [4.0]
(205, 148, 3)
input shape is  torch.Size([3, 205, 148])
WARNING: out of memory
[10/Aug/2019 03:07:44] "POST /rest/1.0/upscale HTTP/1.1" 200 85
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
Input scale is  [3.0]
(205, 148, 3)
input shape is  torch.Size([3, 205, 148])
out put shape is  torch.Size([1, 3, 615, 444])
[10/Aug/2019 03:07:54] "POST /rest/1.0/upscale HTTP/1.1" 200 173166
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
Input scale is  [4.0]
(205, 148, 3)
input shape is  torch.Size([3, 205, 148])
WARNING: out of memory
[10/Aug/2019 03:16:45] "POST /rest/1.0/upscale HTTP/1.1" 200 85
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
Input scale is  [4.0]
(205, 148, 3)
input shape is  torch.Size([3, 205, 148])
WARNING: out of memory
[10/Aug/2019 03:16:48] "POST /rest/1.0/upscale HTTP/1.1" 200 85
<class 'django.core.files.uploadedfile.InMemoryUploadedFile'>
Input scale is  [2.0]
(205, 148, 3)
input shape is  torch.Size([3, 205, 148])
out put shape is  torch.Size([1, 3, 410, 296])
[10/Aug/2019 03:17:03] "POST /rest/1.0/upscale HTTP/1.1" 200 93124