python profiler_python 内存监控模块之memory_profiler

0. memory_profiler是干嘛的

This is a python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for python programs. It is a pure python module and has the psutil module as optional (but highly recommended) dependencies.

memory_profiler是监控python进程的神器，它可以分析出每一行代码所增减的内存状况。

1. 入门例子

#del3.py

importtime

@profiledefmy_func():

a= [1] * (10 ** 6)

b= [2] * (2 * 10 ** 7)

time.sleep(10)delbdelaprint "+++++++++"

if __name__ == '__main__':

my_func()

结果

$python -m memory_profiler del3.py+++++++++Filename: del3.py

Line#Mem usage Increment Line Contents

================================================

2 10.293 MiB 0.000MiB @profile3 defmy_func():4 17.934 MiB 7.641 MiB a = [1] * (10 ** 6)5 170.523 MiB 152.590 MiB b = [2] * (2 * 10 ** 7)6 170.527 MiB 0.004 MiB time.sleep(10)7 17.938 MiB -152.590 MiB delb8 10.305 MiB -7.633 MiB dela9 10.309 MiB 0.004 MiB print "+++++++++"

代码执行一遍，然后给出具体代码在某一步占用的内存，通过内存加减可以看出某个对象的大小。

2. 对象不删除，直接赋值内存是否会继续增长

#对比1

@profiledefmy_func():

a= 'a' * 1024 * 1024 * 1024;

a= 'a' * 1024 * 1024a= 'a' * 1024

delaprint "+++++++++"

if __name__ == '__main__':

my_func()

结果

Line #Mem usage Increment Line Contents

================================================

1 10.293 MiB 0.000MiB @profile2 defmy_func():3 1034.301 MiB 1024.008 MiB a = 'a' * 1024 * 1024 * 1024;4 11.285 MiB -1023.016 MiB a = 'a' * 1024 * 1024

5 11.285 MiB 0.000 MiB a = 'a' * 1024

6 11.285 MiB 0.000 MiB dela7 11.289 MiB 0.004 MiB print "+++++++++"

#对比2

@profiledefmy_func():

a= 'a' * 1024 * 1024 * 1024;dela

a= 'a' * 1024 * 1024

dela

a= 'a' * 1024

delaprint "+++++++++"

if __name__ == '__main__':

my_func()

结果

Line #Mem usage Increment Line Contents

================================================

1 10.293 MiB 0.000MiB @profile2 defmy_func():3 1034.301 MiB 1024.008 MiB a = 'a' * 1024 * 1024 * 1024;4 10.297 MiB -1024.004 MiB dela5 11.285 MiB 0.988 MiB a = 'a' * 1024 * 1024

6 11.285 MiB 0.000 MiB dela7 11.285 MiB 0.000 MiB a = 'a' * 1024

8 11.285 MiB 0.000 MiB dela9 11.289 MiB 0.004 MiB print "+++++++++"

结论：是否 del对象没有影响，新赋的值会替代旧的值

3. 对象赋值是否会增加同样的内存

#对比1

@profiledefmy_func():

a= 'a' * 1024 * 1024 * 1024;

b=adelaprint "+++++++++"

if __name__ == '__main__':

my_func()

结果

Line #Mem usage Increment Line Contents

================================================

1 10.293 MiB 0.000MiB @profile2 defmy_func():3 1034.301 MiB 1024.008 MiB a = 'a' * 1024 * 1024 * 1024;4 1034.301 MiB 0.000 MiB b =a5 1034.301 MiB 0.000 MiB dela6 1034.305 MiB 0.004 MiB print "+++++++++"

#对比2

@profiledefmy_func():

a= 'a' * 1024 * 1024 * 1024;

b=adeladelbprint "+++++++++"

if __name__ == '__main__':

my_func()

结果

Line #Mem usage Increment Line Contents

================================================

1 10.297 MiB 0.000MiB @profile2 defmy_func():3 1034.305 MiB 1024.008 MiB a = 'a' * 1024 * 1024 * 1024;4 1034.305 MiB 0.000 MiB b =a5 1034.305 MiB 0.000 MiB dela6 10.301 MiB -1024.004 MiB delb7 10.305 MiB 0.004 MiB print "+++++++++"

结论，把a赋值给b，内存没有增加。但是只删除其中一个对象的时候，内存不会减。

4. 另一种等价的启动方式

from memory_profiler importprofile

@profile(precision=4)defmy_func():

a= 'a' * 1024 * 1024 * 1024;dela

a= 'a' * 1024 * 1024

dela

a= 'a' * 1024

delaprint "+++++++++"

if __name__ == '__main__':

my_func()

结果

$python -m memory_profiler del3.py+++++++++Filename: del3.py

Line#Mem usage Increment Line Contents

================================================

2 10.3867 MiB 0.0000 MiB @profile(precision=4)3 defmy_func():4 1034.3945 MiB 1024.0078 MiB a = 'a' * 1024 * 1024 * 1024;5 10.3906 MiB -1024.0039 MiB dela6 11.3789 MiB 0.9883 MiB a = 'a' * 1024 * 1024

7 11.3789 MiB 0.0000 MiB dela8 11.3789 MiB 0.0000 MiB a = 'a' * 1024

9 11.3789 MiB 0.0000 MiB dela10 11.3828 MiB 0.0039 MiB print "+++++++++"

5. 非python内置对象例子

from memory_profiler importprofileimportnetworkx as nx

@profile(precision=4)defmy_func():

a= 'a' * 1024 * 1024 * 1024;dela

G=nx.Graph()

G.add_node(1)

G.add_nodes_from([ifor i in range(10000)])

G.add_nodes_from([ifor i in range(10000, 20000)])

G.add_edges_from([(1,2), (1,4), (2, 9), (4, 1), (3, 8)])delGprint "++++++"

if __name__ == '__main__':

my_func()

结果

$python del3.py++++++Filename: del3.py

Line#Mem usage Increment Line Contents

================================================

4 23.4844 MiB 0.0000 MiB @profile(precision=4)5 defmy_func():6 1047.4922 MiB 1024.0078 MiB a = 'a' * 1024 * 1024 * 1024;7 23.4883 MiB -1024.0039 MiB dela8 23.4883 MiB 0.0000 MiB G =nx.Graph()9 23.4883 MiB 0.0000 MiB G.add_node(1)10 31.3359 MiB 7.8477 MiB G.add_nodes_from([i for i in range(10000)])11 36.9219 MiB 5.5859 MiB G.add_nodes_from([i for i in range(10000, 20000)])12 36.9219 MiB 0.0000 MiB G.add_edges_from([(1,2), (1,4), (2, 9), (4, 1), (3, 8)])13 25.9219 MiB -11.0000 MiB delG14 25.9258 MiB 0.0039 MiB print "++++++"

6. 类怎么使用呢

#del4.py

from memory_profiler importprofileclasspeople:

name= ''age=0__weight =0def __init__(self,n,a,w):

self.name=n

self.age=a

self.__weight =w

@profile(precision=4)defspeak(self):

a= 'a' * 1024b= 'b' * 1024 * 1024

print("%s is speaking: I am %d years old" %(self.name,self.age))if __name__ == '__main__':

p= people('tom', 10, 30)

p.speak()

结果

$python del4.py

tomis speaking: I am 10years old

Filename: del4.py

Line#Mem usage Increment Line Contents

================================================

13 9.4219 MiB 0.0000 MiB @profile(precision=4)14 defspeak(self):15 9.4258 MiB 0.0039 MiB a = 'a' * 1024

16 10.4297 MiB 1.0039 MiB b = 'b' * 1024 * 1024

17 10.4336 MiB 0.0039 MiB print("%s is speaking: I am %d years old" % (self.name,self.age))

7. 随时间内存统计

#test.py

importtime

@profiledeftest1():

n= 10000a= [1] *n

time.sleep(1)returna

@profiledeftest2():

n= 100000b= [1] *n

time.sleep(1)returnbif __name__ == "__main__":

test1()

test2()

test.py 里有两个两个待分析的函数(@profile标识)，为了形象地看出内存随时间的变化，每个函数内sleep 1s，执行

mprof run test.py

如果执行成功，结果这样

$ mprof run test.py

mprof: Sampling memory every 0.1s

running as a Python program...

结果会生成一个.dat文件，如"mprofile_20160716170529.dat"，里面记录了内存随时间的变化，可用下面的命令以图片的形式展示出来：

mprof plot

8. API

memory_profiler提供很多包给第三方代码，如

>>> from memory_profiler importmemory_usage>>> mem_usage = memory_usage(-1, interval=.2, timeout=1)>>> print(mem_usage)

[7.296875, 7.296875, 7.296875, 7.296875, 7.296875]

memory_usage(proc=-1, interval=.2, timeout=None)返回一段时间的内存值，其中proc=-1表示此进程，这里可以指定特定的进程号；interval=.2表示监控的时间间隔是0.2秒；timeout=1表示总共的时间段为1秒。那结果就返回5个值。

如果要返回一个函数的内存消耗，示例

def f(a, n=100):importtime

time.sleep(2)

b= [a] *n

time.sleep(1)returnbfrom memory_profiler importmemory_usageprint memory_usage((f, (2,), {'n' : int(1e6)}))

这里执行了 f(1, n=int(1e6)) ，并返回在执行此函数时的内存消耗。

9. 优化实例

对比str & int

from datetime import datetime

@profile

def my_func():

beg=datetime.now()

a={}for i in range(1000000):

a[i]=i

#a[str(i)]=i

print"+++++++++"del a

print"+++++++++"end=datetime.now()

print"time:", end -begif __name__ == '__main__':

my_func()

用a[i] = i，结果

+++++++++

+++++++++time: 0:06:14.790899Filename: int.py

Line#Mem usage Increment Line Contents

================================================

2 14.727 MiB 0.000MiB @profile3 defmy_func():4 14.734 MiB 0.008 MiB beg =datetime.now()5 14.734 MiB 0.000 MiB a ={}6 94.031 MiB 79.297 MiB for i in range(1000000):7 94.031 MiB 0.000 MiB a[i] =i8 #a[str(i)] = i

9 86.402 MiB -7.629 MiB print "+++++++++"

10 38.398 MiB -48.004 MiB dela11 38.398 MiB 0.000 MiB print "+++++++++"

12 38.398 MiB 0.000 MiB end =datetime.now()13 38.406 MiB 0.008 MiB print "time:", end - beg

用a[str(i)] = i，结果

+++++++++

+++++++++time: 0:06:00.288052Filename: int.py

Line#Mem usage Increment Line Contents

================================================

2 14.723 MiB 0.000MiB @profile3 defmy_func():4 14.730 MiB 0.008 MiB beg =datetime.now()5 14.730 MiB 0.000 MiB a ={}6 140.500 MiB 125.770 MiB for i in range(1000000):7 #a[i] = i

8 140.500 MiB 0.000 MiB a[str(i)] =i9 132.871 MiB -7.629 MiB print "+++++++++"

10 38.539 MiB -94.332 MiB dela11 38.539 MiB 0.000 MiB print "+++++++++"

12 38.539 MiB 0.000 MiB end =datetime.now()13 38.547 MiB 0.008 MiB print "time:", end - beg