多线程 vs 多进程
-
程序:一堆代码以文本形式存入一个文档
-
进程:程序运行的一个状态 (一套房子/包工头)
- 包含地址空间、内存、数据栈等
- 每个进程由自己完全独立的运行环境,多进程共享数据是一个问题
-
线程 (一套房子中的一个房间/一个工人)
- 一个线程的独立运行片段,一个进程可以有多个线程
- 轻量化的进程
- 一个进程的多个线程间共享数据和上下文运行环境
- 共享互斥问题
-
全局解释器锁(GIL)
- Python代码的执行是由Python虚拟机进行控制
- 在主循环中只能有一个控制线程在执行
-
Python包
- thread: 有问题,不好用,python3改成了_thread
- threading:通行的包
- 案例01 顺序执行,耗时比较长
- 案例02 改用多线程,缩短总时间,使用_thread
- 案例03 多线程,传参数
# 案例01
# 利用time函数,生成两个函数,顺序调用,计算总运行时间
import time
def loop1():
print('Start loop 1 at:',time.ctime())
time.sleep(4)
print("Ens loop 1 at:",time.ctime())
def loop2():
print('Start loop 2 at:',time.ctime())
time.sleep(2)
print("Ens loop 2 at:",time.ctime())
def main():
print("Start at:",time.ctime())
loop1()
loop2()
print("All done at:",time.ctime())
if __name__ == "__main__":
main()
Start at: Tue Mar 17 15:49:19 2020
Start loop 1 at: Tue Mar 17 15:49:19 2020
Ens loop 1 at: Tue Mar 17 15:49:23 2020
Start loop 2 at: Tue Mar 17 15:49:23 2020
Ens loop 2 at: Tue Mar 17 15:49:25 2020
All done at: Tue Mar 17 15:49:25 2020
# 案例02
import time
import _thread as thread
def loop1():
print('Start loop 1 at:',time.ctime())
time.sleep(4)
print("Ens loop 1 at:",time.ctime())
def loop2():
print('Start loop 2 at:',time.ctime())
time.sleep(2)
print("Ens loop 2 at:",time.ctime())
def main():
print("Start at:",time.ctime())
thread.start_new_thread(loop1,())#hou后面括号是放一个参数,没有参数也得写()
thread.start_new_thread(loop2,())
print("All done at:",time.ctime())
if __name__ == "__main__":
#一定要有while语句,不然主线程执行完毕,则子线程可能也需要终止
while True:
time.sleep(1)
Start at: Tue Mar 17 16:05:34 2020
All done at: Tue Mar 17 16:05:34 2020
Start loop 1 at:Start loop 2 at: Tue Mar 17 16:05:34 2020
Tue Mar 17 16:05:34 2020
Ens loop 2 at: Tue Mar 17 16:05:36 2020
Ens loop 1 at: Tue Mar 17 16:05:38 2020
# 案例03
import time
import _thread as thread
def loop1(in1):
print('Start loop 1 at:',time.ctime())
print("我是参数",in1)
time.sleep(4)
print("Ens loop 1 at:",time.ctime())
def loop2(in1,in2):
print('Start loop 2 at:',time.ctime())
print("我是参数",in1,"和参数",in2)
time.sleep(2)
print("Ens loop 2 at:",time.ctime())
def main():
print("Start at:",time.ctime())
thread.start_new_thread(loop1,("王老大",))# 是元组,所以后面需要加逗号
thread.start_new_thread(loop2,("谢老大","烨老大"))
print("All done at:",time.ctime())
if __name__ == "__main__":
main()
while True:
time.sleep(10)
Start at: Tue Mar 17 16:52:22 2020
All done at: Tue Mar 17 16:52:22 2020
Start loop 2 at: Tue Mar 17 16:52:22 2020
我是参数 谢老大 和参数 烨老大
Start loop 1 at: Tue Mar 17 16:52:22 2020
我是参数 王老大
Ens loop 2 at: Tue Mar 17 16:52:24 2020
Ens loop 1 at: Tue Mar 17 16:52:26 2020
threading的使用
- 直接利用threading.Thread生成Thread实例
1. t = threading.Thread(target=xxx,args=(xxx,))
2. t.start(): 启动多线程
3. t.join(): 等待多线程执行完成
import time
import threading
def loop1(in1):
print('Start loop 1 at:',time.ctime())
print("我是参数",in1)
time.sleep(4)
print("Ens loop 1 at:",time.ctime())
def loop2(in1,in2):
print('Start loop 2 at:',time.ctime())
print("我是参数",in1,"和参数",in2)
time.sleep(2)
print("Ens loop 2 at:",time.ctime())
def main():
print("Start at:",time.ctime())
t1 = threading.Thread(target=loop1,args=("王老大",))# 是元组,所以后面需要加逗号
t1.start()
t2 = threading.Thread(target=loop2,args=("谢老大","宗老大"))
t2.start()
print("All done at:",time.ctime())
if __name__ == "__main__":
main()
# 死循环
while True:
time.sleep(20)
Start at: Wed Mar 18 16:35:47 2020
Start loop 1 at:Start loop 2 at: All done at: Wed Mar 18 16:35:47 2020
Wed Mar 18 16:35:47 2020
我是参数 王老大
Wed Mar 18 16:35:47 2020
我是参数 谢老大 和参数 宗老大
Ens loop 2 at: Wed Mar 18 16:35:49 2020
Ens loop 1 at: Wed Mar 18 16:35:51 2020
# 加入join后比较
import time
import threading
def loop1(in1):
print('Start loop 1 at:',time.ctime())
print("我是参数",in1)
time.sleep(4)
print("Ens loop 1 at:",time.ctime())
def loop2(in1,in2):
print('Start loop 2 at:',time.ctime())
print("我是参数",in1,"和参数",in2)
time.sleep(2)
print("Ens loop 2 at:",time.ctime())
def main():
print("Start at:",time.ctime())
t1 = threading.Thread(target=loop1,args=("王老大",))# 是元组,所以后面需要加逗号
t1.start()
t2 = threading.Thread(target=loop2,args=("谢老大","宗老大"))
t2.start()
t1.join()
t2.join()
print("All done at:",time.ctime())
if __name__ == "__main__":
main()
while True:
time.sleep(10)
Start at: Tue Mar 17 17:43:50 2020
Start loop 1 at:Start loop 2 at: Tue Mar 17 17:43:50 2020
我是参数 王老大
Tue Mar 17 17:43:50 2020
我是参数 谢老大 和参数 宗老大
Ens loop 2 at: Tue Mar 17 17:43:52 2020
Ens loop 1 at: Tue Mar 17 17:43:54 2020
All done at: Tue Mar 17 17:43:54 2020
守护线程-daemon
- 如果在程序中将子线程设置成守护线程,则子线程会在主线程结束的时候自动退出
- 一般认为,守护线程不重要或者不允许离开主线程独立运行
- 守护线程案例能否有效果跟环境相关
# 非守护线程
import time
import threading
def fun():
print("Start fun")
time.sleep(2)
print("End fun")
print("Main thread")
t1 = threading.Thread(target=fun,args=())
t1.start()
time.sleep(1)
print("Main thread end")
Main thread
Start fun
Main thread end
End fun
# 守护线程,jupyter可能会打印出来最后一句,pycharm不会,跟环境有关
import time
import threading
def fun():
print("Start fun")
time.sleep(2)
print("End fun")
print("Main thread")
t1 = threading.Thread(target=fun,args=())
#守护线程的方法,必须在start之前设置,否则无效
t1.setDaemon(True)
# t1.Daemon = True
t1.start()
time.sleep(1)
print("Main thread end")
Main thread
Start fun
Main thread end
End fun
线程常用属性
- threading.currentThread: 返回当前线程变量
- threading.enumerate: 返回一个包含正在运行的线程的list,正在运行的线程指的是线程启动后运行的线程
- threading.activeCount: 返回正在运行的线程数量,效果跟len(threading.enumerate)相似
- thr.setName: 给线程设置名字
- thr.getName: 得到线程的名字
import time
import threading
def loop1():
print('Start loop 1 at:',time.ctime())
time.sleep(5)
print("Ens loop 1 at:",time.ctime())
def loop2():
print('Start loop 2 at:',time.ctime())
time.sleep(2)
print("Ens loop 2 at:",time.ctime())
def loop3():
print('Start loop 3 at:',time.ctime())
time.sleep(6)
print("Ens loop 3 at:",time.ctime())
def main():
print("Starting at:",time.ctime())
t1 = threading.Thread(target=loop1,args=())
t1.setName('THREAD_1')
t1.start()
t2 = threading.Thread(target=loop2,args=())
t2.setName('THREAD_2')
t2.start()
t3 = threading.Thread(target=loop3,args=())
t3.setName('THREAD_3')
t3.start()
#预计三秒后,thread2已经自动结束
time.sleep(3)
for thr in threading.enumerate():
print("正在运行的子线程名字是:{0}".format(thr.getName()))
print("正在运行子线程数量为:{0}",format(threading.activeCount()))
print("All done at:",time.ctime())
if __name__ == "__main__":
main()
while True:
time.sleep(10)
Starting at: Wed Mar 18 09:23:07 2020
Start loop 1 at: Wed Mar 18 09:23:07 2020
Start loop 2 at:Start loop 3 at: Wed Mar 18 09:23:07 2020
Wed Mar 18 09:23:07 2020
Ens loop 2 at: Wed Mar 18 09:23:09 2020
正在运行的子线程名字是:MainThread
正在运行的子线程名字是:Thread-4
正在运行的子线程名字是:Thread-5
正在运行的子线程名字是:IPythonHistorySavingThread
正在运行的子线程名字是:Thread-3
正在运行的子线程名字是:THREAD_1
正在运行的子线程名字是:THREAD_3
正在运行子线程数量为:{0} 7
All done at: Wed Mar 18 09:23:10 2020
Ens loop 1 at: Wed Mar 18 09:23:12 2020
Ens loop 3 at: Wed Mar 18 09:23:13 2020
直接继承自threading.Thread
- 直接继承Thread
- 重写run函数
- 类实例可以直接运行
import time
import threading
# 1.类需要继承自threading.Thread
class MyThread(threading.Thread):
def __init__(self,arg):
super(MyThread,self).__init__()
self.arg = arg
# 必须重写run函数。run函数代表的是真正执行的功能
def run(self):
time.sleep(2)
print("The args for this class is {}".format(self.arg))
for i in range(5):
t = MyThread(i)
t.start()
t.join()
print("Main thread is done!!!!!!")
The args for this class is 0
The args for this class is 1
The args for this class is 2
The args for this class is 3
The args for this class is 4
Main thread is done!!!
共享变量
- 共享变量:当多个线程同时访问一个变量的时候,会产生共享变量的问题
# 一般情况是这样
import threading
sum = 0
loopSum = 1000000
def myAdd():
global sum,loopSum
for i in range(1,loopSum):
sum += 1
def myMinu():
global sum,loopSum
for i in range(1,loopSum):
sum -= 1
if __name__ == "__main__":
myAdd()
print(sum)
myMinu()
print(sum)
999999
0
# 共享变量同时进行,就会混淆,每次执行的结果都不一样
import threading
sum = 0
loopSum = 1000000
def myAdd():
global sum,loopSum
for i in range(1,loopSum):
sum += 1
def myMinu():
global sum,loopSum
for i in range(1,loopSum):
sum -= 1
if __name__ == "__main__":
t1 = threading.Thread(target=myAdd,args=())
t2 = threading.Thread(target=myMinu,args=())
t1.start()
t2.start()
t1.join()
t2.join()
print("Done ....{0}".format(sum))
Done …90917
解决这个问题
- 锁,信号灯
- 锁(Lock):
- 是一个标志,表示一个线程在占用一些资源
- 使用方法
- 上锁
- 使用共享资源,放心的用
- 取消锁,释放锁
- 锁(Lock):
- 锁谁:哪个资源需要多个线程共享,锁哪个
- 理解锁:锁其实不是锁住谁,而是一个令牌
import threading
sum = 0
loopSum = 1000000
lock = threading.Lock()
def myAdd():
global sum, loopSum
for i in range(1,loopSum):
# 上锁,申请锁
lock.acquire()
sum += 1
#释放锁
lock.release()
def myMinu():
global sum,loopSum
for i in range(1,loopSum):
lock.acquire()
sum -= 1
lock.release()
if __name__ == "__main__":
t1 = threading.Thread(target=myAdd,args=())
t2 = threading.Thread(target=myMinu,args=())
t1.start()
t2.start()
t1.join()
t2.join()
print("Done ....{0}".format(sum))
Done …0
线程安全问题
- 如果一个资源/变量,它对于多线程来讲,不用加锁也不会引起任何问题,则称为线程安全
- 线程不安全变量类型:list,set,dict
- 线程安全变量类型:queue
生产者消费者问题
- 一个模型,可以用来搭建消息队列
- queue是一个用来存放变量的数据结构,特点是先进先出,内部元素排队,可以理解成一个特殊的list
import threading
import time
import queue
class Producer(threading.Thread):
def run(self):
global queue
count = 0
while True:
if queue.qsize() < 1000:#队列的长度
for i in range(100):
count = count +1
msg = "生成产品"+str(count)
queue.put(msg)#生成的值放入队列
print(msg)
time.sleep(0.5)
class Consumer(threading.Thread):
def run(self):
global queue
while True:
if queue.qsize() >100:#队列的长度
for i in range(3):
msg = self.name + "消费了" + queue.get()
print(msg)
time.sleep(1)
if __name__ == "__main__":
queue = queue.Queue()
for i in range(500):
queue.put('初始产品'+str(i))
for i in range(2):
p = Producer()
p.start()
for i in range(5):
c = Consumer()
c.start()
死锁问题
- 两个锁都在同时执行,申请并同时执行了,甲方就在申请不到乙方的锁
- 解决死锁的问题,只有平时写代码注意
import threading
import time
lock_1 = threading.Lock()
lock_2 = threading.Lock()
def func_1():
print("func_1 starting.......")
lock_1.acquire()
print("func_1 申请了 lock_1...")
time.sleep(2)
print("func_1 等待 lock_2")
lock_2.acquire()
print("func_1 申请了 lock_2")
lock_2.release()
print("func_1 释放了 lock_2")
lock_1.release()
print("func_1 释放了 lock_1")
print("fun_1 done.....")
def func_2():
print("func_2 starting.......")
lock_2.acquire()
print("func_2 申请了 lock_2...")
time.sleep(4)
print("func_2 等待 lock_1")
lock_1.acquire()
print("func_2 申请了 lock_1")
lock_1.release()
print("func_2 释放了 lock_1")
lock_2.release()
print("func_2 释放了 lock_2")
print("fun_2 done.....")
if __name__ == "__main__":
print("主程序启动。。。。。")
t1 = threading.Thread(target=func_1,args=())
t2 = threading.Thread(target=func_2,args=())
t1.start()
t2.start()
t1.join()
t2.join()
print("主程序结束。。。。。")
主程序启动。。。。。
func_1 starting…
func_1 申请了 lock_1…
func_2 starting…
func_2 申请了 lock_2…
func_1 等待 lock_2
func_2 等待 lock_1
semaphore(信号灯)
- 允许一个资源最多由几个多线程同时使用
import threading
import time
# 参数定义最多定义几个线程同时使用资源
# 678执行了,6出去9进来,7出去10进来,依次交替
semaphore = threading.Semaphore(3)
def func():
if semaphore.acquire():
for i in range(5):
print(threading.currentThread().getName()+"get semaphore")
time.sleep(15)
semaphore.release()
print(threading.currentThread().getName()+"release semaphore")
for i in range(8):
t1 = threading.Thread(target=func)
t1.start()
threading.Timer
- Timer是利用多线程,在指定时间后启动一个功能
import threading
import time
def func():
print("I am running......")
time.sleep(4)
print("I am done......")
if __name__ == "__main__":
t = threading.Timer(6,func)#6秒后再执行func
t.start()
i = 0
while True:
print("{0}*8*8**888".format((i)))
time.sleep(3)
i += 1
可重入锁
- 一个锁,可以被一个线程多次申请
- 主要解决递归调用的时候,需要申请锁的情况
import threading
import time
class MyThread(threading.Thread):
def run(self):
global num
time.sleep(1)
num = 0
mutex = threading.Thread()
if mutex.acquire(1):
num +=1
msg = self.name+"set num to"+str(num)
print(msg)
mutex.acquire()
mutex.release()
mutex.release()
def testTh():
for i in range(5):
t = MyThread()
t.start()
if __name__ == "__main__":
testTh()
#不清楚出错的原因,可能是环境的问题,技术没到家不晓得
线程替代方案
- subprocess
- 完全跳过线程,使用进程
- 是Python进程的主要替代方案
- python2.4后引入
- multiprocessing
- 使用threadinhg接口Python,使用子进程
- 允许为多核或者多CPU Python进程
- python2.6
- concurrent.futures
- 新的异步执行模块
- 任务级别的操作
- python3.2后引入