python re入门——从训练记录log文件获得关键信息（附完整代码）

问题描述

我们在使用深度学习训练模型和做其他一些处理的时候，往往需要记录log文件，保存配置信息和一部分结果。本文基于re实现从不同的log文件中，获取不同样本的PSNR，并求出PSNR的平均值。
log文件的内容如下图所示，我们需要获取不同的样本序号（例如25591）和对应的PSNR值。

                            < M A T L A B (R) >
                  Copyright 1984-2017 The MathWorks, Inc.
                   R2017a (9.2.0.538062) 64-bit (glnxa64)
                             February 23, 2017

 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
./test/RCAN/Recon_SGD_100mid30/Reconx2_LR_25589.mat
/home/usrname/project/ReconNet/dataset_GT3D/Xguess_direct/Xguess_25589.mat
single
single
最大值最小值
  2.0833e+04

     0

  1.3346e+04

     0

差值的绝对值的和
    90656192

PSNR is 40.5706
25589 get psnr over

                            < M A T L A B (R) >
                  Copyright 1984-2017 The MathWorks, Inc.
                   R2017a (9.2.0.538062) 64-bit (glnxa64)
                             February 23, 2017

 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
./test/RCAN/Recon_SGD_100mid30/Reconx2_LR_25590.mat
/home/usrname/project/ReconNet/dataset_GT3D/Xguess_direct/Xguess_25590.mat
single
single
最大值最小值
  1.5590e+04

     0

  9.8991e+03

     0

差值的绝对值的和
    66605736

PSNR is 38.3033
25590 get psnr over

                            < M A T L A B (R) >
                  Copyright 1984-2017 The MathWorks, Inc.
                   R2017a (9.2.0.538062) 64-bit (glnxa64)
                             February 23, 2017

 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
./test/RCAN/Recon_SGD_100mid30/Reconx2_LR_25591.mat
/home/usrname/project/ReconNet/dataset_GT3D/Xguess_direct/Xguess_25591.mat
single
single
最大值最小值
  1.4285e+04

     0

  9.4557e+03

     0

差值的绝对值的和
    71322160

PSNR is 37.5057
25591 get psnr over

                            < M A T L A B (R) >
                  Copyright 1984-2017 The MathWorks, Inc.
                   R2017a (9.2.0.538062) 64-bit (glnxa64)
                             February 23, 2017

 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
./test/RCAN/Recon_SGD_100mid30/Reconx2_LR_25592.mat
/home/usrname/project/ReconNet/dataset_GT3D/Xguess_direct/Xguess_25592.mat
single
single
最大值最小值
  1.8736e+04

     0

  1.5317e+04

     0

差值的绝对值的和
   167848192

PSNR is 33.433
25592 get psnr over

                            < M A T L A B (R) >
                  Copyright 1984-2017 The MathWorks, Inc.
                   R2017a (9.2.0.538062) 64-bit (glnxa64)
                             February 23, 2017

 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
./test/RCAN/Recon_SGD_100mid30/Reconx2_LR_25593.mat
/home/usrname/project/ReconNet/dataset_GT3D/Xguess_direct/Xguess_25593.mat
single
single
最大值最小值
  1.8384e+04

     0

  1.4160e+04

     0

差值的绝对值的和
   159005152

PSNR is 33.9701
25593 get psnr over

                            < M A T L A B (R) >
                  Copyright 1984-2017 The MathWorks, Inc.
                   R2017a (9.2.0.538062) 64-bit (glnxa64)
                             February 23, 2017

 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
./test/RCAN/Recon_SGD_100mid30/Reconx2_LR_25594.mat
/home/usrname/project/ReconNet/dataset_GT3D/Xguess_direct/Xguess_25594.mat
single
single
最大值最小值
  2.2593e+04

     0

  1.5102e+04

     0

差值的绝对值的和
   260633296

PSNR is 32.6955
25594 get psnr over

问题解决

我们使用按行读取log文本的内容，并去除’\n’

with open(logName, 'r') as f1:
    list1 = f1.readlines()

for i in range(0, len(list1)):
    list1[i] = list1[i].rstrip('\n')

然后使用findall在字符串中找到正则表达式所匹配的所有子串，并返回一个列表，如果没有找到匹配的，则返回空列表。

import re 

分别获取以下语句：
PSNR is 38.2274
25592 get psnr

a = re.findall('^PSNR is [0-9.?0-9]*',list1[i])
b = re.findall('^[0-9]* get psnr',list1[i])

其中？表示出现0或1次，*表示出现0-无穷次。结果a和b都是列表，当它不为空说明提取到了上面的语句。
然后在a[0]和b[0]中提取数字，也就是找到38.2274和25592：

        if len(a): 
            # print(a)
            val = re.findall('[0-9.?0-9]+',a[0])

 	    if len(b):
            # print(b)
            key = re.findall('^[0-9]+',b[0])

+表示出现1到无穷次。我们可以将样本序号作为键，对应的PSNR作为值保存在字典中。

完整代码

import re
'''
@Author: AsajuHuishi
@Date: 20/11/1
'''

def getResult(logName, dictName):
    with open(logName, 'r') as f1:
        list1 = f1.readlines()

    for i in range(0, len(list1)):
        list1[i] = list1[i].rstrip('\n')
        # print(list1[i])
        a = re.findall('^PSNR is [0-9.?0-9]*',list1[i])
        if len(a): 
            # print(a)
            val = re.findall('[0-9.?0-9]+',a[0])
            # print('PSNR: ',val[0])
            
        b = re.findall('^[0-9]* get psnr',list1[i])
        if len(b):
            # print(b)
            key = re.findall('^[0-9]+',b[0])
            # print('mat: ',key[0])
            dictName[key[0]] = val[0]
            
    print('psnr字典', dictName, '内含测试样本数量', len(dictName))
    print('平均值',(sum([float(i) for i in list(dictName.values())])/float(len(dictName))))
    return dictName

def main():
    psnr3d = dict()
    psnr4d = dict()
    print('3d')
    ret3D = getResult('pp3D_20.log',psnr3d)
    print('4d')
    ret4D = getResult('pp4D_20.log',psnr4d)
    
if __name__ == "__main__":
    main()