python多线程队列文件_在内部线程中调用时，Python队列内存泄漏

I have python TCP client and need to send media(.mpg) file in a loop to a 'C' TCP server.

I have following code, where in separate thread I am reading the 10K blocks of file and sending it and doing it all over again in loop, I think it is because of my implementation of thread module, or tcp send. I am using Queues to print the logs on my GUI ( Tkinter ) but after some times it goes out of memory..

UPDATE 1 - Added more code as requested

Thread class "Sendmpgthread" used to create thread to send data

.

.

def __init__ ( self, otherparams,MainGUI):

.

.

self.MainGUI = MainGUI

self.lock = threading.Lock()

Thread.__init__(self)

#This is the one causing leak, this is called inside loop

def pushlog(self,msg):

self.MainGUI.queuelog.put(msg)

def send(self, mysocket, block):

size = len(block)

pos = 0;

while size > 0:

try:

curpos = mysocket.send(block[pos:])

except socket.timeout, msg:

if self.over:

self.pushlog(Exit Send)

return False

except socket.error, msg:

print 'Exception'

return False

pos = pos + curpos

size = size - curpos

return True

def run(self):

media_file = None

mysocket = None

try:

mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

mysocket.connect((self.ip, string.atoi(self.port)))

media_file = open(self.file, 'rb')

while not self.over:

chunk = media_file.read(10000)

if not chunk: # EOF Reset it

print 'resetting stream'

media_file.seek(0, 0)

continue

if not self.send(mysocket, chunk): # If some error or thread is killed

break;

#disabling this solves the issue

self.pushlog('print how much data sent')

except socket.error, msg:

print 'print exception'

except Exception, msg:

print 'print exception'

try:

if media_file is not None:

media_file.close()

media_file = None

if mysocket is not None:

mysocket.close()

mysocket = None

finally:

print 'some cleaning'

def kill(self):

self.over = True

I figured out that it is because of wrong implementation of Queue as commenting that piece resolves the issue

UPDATE 2 - MainGUI class which is called from above Thread class

class MainGUI(Frame):

def __init__(self, other args):

#some code

.

.

#from the above thread class used to send data

self.send_mpg_status = Sendmpgthread(params)

self.send_mpg_status.start()

self.after(100, self.updatelog)

self.queuelog = Queue.Queue()

def updatelog(self):

try:

msg = self.queuelog.get_nowait()

while msg is not None:

self.printlog(msg)

msg = self.queuelog.get_nowait()

except Queue.Empty:

pass

if self.send_mpg_status: # only continue when sending

self.after(100, self.updatelog)

def printlog(self,msg):

#print in GUI

解决方案

Since printlog is adding to a tkinter text control, the memory occupied by that control will grow with each message (it has to store all the log messages in order to display them).

Unless storing all the logs is critical, a common solution is to limit the maximum number of log lines displayed.

A naive implementation is to eliminate extra lines from the begining after the control reaches a maximum number of messages. Add a function to get the number of lines in the control and then, in printlog something similar to:

while getnumlines(self.edit) > self.maxloglines:

self.edit.delete('1.0', '1.end')

(above code not tested)

update: some general guidelines

Keep in mind that what might look like a memory leak does not always mean that a function is wrong, or that the memory is no longer accessible. Many times there is missing cleanup code for a container that is accumulating elements.

A basic general approach for this kind of problems:

form an opinion on what part of the code might be causing the problem

check it by commenting that code out (or keep commenting code until you find a candidate)

look for containers in the responsible code, add code to print their size

decide what elements can be safely removed from that container, and when to do it

test the result