新手写了一个多线程的爬虫,所有线程都执行完了,但是一直占着 1.5GB 的内存(任务数越多不释放的内存越多)
不知道怎么排查哪里出问题,pympler 看不太懂问题到底出在哪里,请教该如何正确的排查问题
执行多线程函数的代码:
def mainfunc(tasknum, thread):
tr = tracker.SummaryTracker()
tr.print_diff()
list = []
for i in range(tasknum):
list.append(str(i))
pool = threadpool.ThreadPool(thread)
requests = threadpool.makeRequests(childfunc, list)
for req in requests:
pool.putRequest(req)
pool.wait()
tr.print_diff()
tr.print_diff()打印的内容
初始化:
types | # objects | total size
========================== | =========== | ============
list | 3741 | 350.84 KB
str | 3739 | 260.01 KB