在写爬虫中,我要把爬取到的数据存到数据库中.每一个页面里边有很多条目,比如一个人的访客可能有很多个,于是插入卸载循环中,
try:
sql_visitor='INSERT INTO visitor (ownername,owneruid,visitorname,visitoruid,visittime) VALUE ("%s",%d,"%s",%d,"%s")'%(ownername,owneruid,visitorname,visitoruid,visitortime)
print sql_visitor
self.cursor.execute(sql_visitor)
self.connect.commit()
except Exception as e:
print e
一个页面一个线程,嫌弃慢的我开了5个
max_threads=5
while uid < 8000000 or threadlist:
for thread1 in threadlist:
if not thread1.is_alive():
threadlist.remove(thread1)
while len(threadlist) < max_threads and uid < 8000000:
uid+=1
thread2=threading.Thread(target=run,args=(uid,))
thread2.setDaemon(True)
thread2.start()
threadlist.append(thread2)
time.sleep(5)
运行很顺利:
INSERT