卷积神经网络的训练经常耗时很长,为了加速训练提高炼丹效率,首先需要定位代码里哪些地方耗时长,比如数据预处理时间长,可以试试nvidia的dali加速。line_profiler库可以分析每一行代码的运行时间,方便定位程序运行效率瓶颈。
安装
pip install line_profiler
使用
参考这篇博文
测试代码1:
from line_profiler import LineProfiler
import random
def do_stuff(numbers):
s = sum(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
if __name__=='__main__':
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
简而言之,我们需要运行的函数加上一个包装,把参数赋给包装会自动执行函数,最后调用打印统计信息的函数即可。
可以看到如下的输出,显示了每一行代码的运行时间和占用百分比
当然,我们经常会在函数内调用其他函数,可以使用add_function解决,修改代码1得到如下的代码2:
from line_profiler import LineProfiler
import random
def do_one_stuff(numbers):
l = [numbers[i]/43 for i in range(len(numbers))]
def do_other_stuff(numbers):
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
def do_stuff(numbers):
for i in range(3):
print(i)
s = sum(numbers)
do_one_stuff(numbers)
do_other_stuff(numbers)
if __name__=='__main__':
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp.add_function(do_one_stuff)
lp.add_function(do_other_stuff)
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
此时的输出结果如下图,可以看到每个被添加的函数都会显示该函数内每行代码执行时间:
(引入其他文件的函数时,与使用本文件中定义的函数同理,详见下一段代码中被注释的代码段)
下面的代码3分析了卷积神经网络CNN模型test阶段时间分布:
from line_profiler import LineProfiler
import random
import os
import torch
from modeling.csrnet import CSRNet
from main_csrnet import get_loader
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from time_analysis import do_stuff, do_one_stuff, do_other_stuff
def val(model, test_loader):
model.eval()
mae,mse = 0.0,0.0
with torch.no_grad():
for img, _, count in test_loader:
img = img.cuda()
output = model(img)
est = output.sum().item()
mae += abs(est - count.item())
mse += (est - count.item())**2
mae /= len(test_loader)
mse /= len(test_loader)
mse = mse**0.5
return mae, mse
def main():
net = CSRNet(upsample=False, bn=False)
net.cuda()
train_path = '/home/datamining/Datasets/CrowdCounting/shanghai/part_A_final/train_data/images/'
test_path = '/home/datamining/Datasets/CrowdCounting/shanghai/part_A_final/test_data/images/'
train_loader, test_loader = get_loader(train_path, test_path, 8)
best_mae, best_mse = val(net, test_loader)
if __name__=='__main__':
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp.add_function(val)
lp_wrapper = lp(main)
lp_wrapper()
'''
lp.add_function(do_one_stuff)
lp.add_function(do_other_stuff)
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
'''
lp.print_stats()
可以看到output = model(img)并不是用时最长的,反而是est = output.sum().item()这一行时间最长。
for img, _, count in test_loader: 这一行和img = img.cuda()这一行用了挺长的时间,说明数据的读取和存到gpu上的过程可能还需要优化。
类似的分析工具还有cProfile
最后,还没发现怎么用md模式,感觉写起来不习惯。