Python学习整理记录之一26.线程与进程

1、关于概念的收集和整理

• 程序：一堆代码以文本形式存入一个文档
• 进程：程序运行的一个状态
  • 包含地址空间、内存、数据栈等
  • 每个进程由自己完全独立的运行环境，多进程共享数据是一个问题
• 线程：
  • 一个进程的独立运行片段，一个进程可以由多个线程
  • 轻量化的进程
  • 一个进程的多个现成间共享数据和上下文运行环境
  • 共享互斥问题
• 全局解释器锁（GIL）
  • Python代码的执行是由Python虚拟机进行控制
  • 在主循环中只能有一个控制线程在执行

2、关于函数和方法使用的收集整理

• Python的包
  • thread：P3改成_thread
    • 案例01，02，03

# 01 这是一个单线程的执行代码
import time
def loop1():
    print("Start loop 1 at :",time.ctime())
    time.sleep(4)
    print("End loop 1 at: ", time.ctime())
def loop2():
    print("Start loop 2 at :",time.ctime())
    time.sleep(2)
    print("End loop 2 at: ", time.ctime())
def main():
    print("Starting at: ", time.ctime())
    loop1()
    loop2()
    print("All done at:", time.ctime())
main()

# 02
import time
import _thread as thread
def loop1():
    print("Start loop 1 at :",time.ctime())
    time.sleep(4)
    print("End loop 1 at: ", time.ctime())
def loop2():
    print("Start loop 2 at :",time.ctime())
    time.sleep(2)
    print("End loop 2 at: ", time.ctime())
def main():
    print("Starting at: ", time.ctime())
    # 启动多线程的意思是用多线程去执行某个函数
    # 启动函数：start_new_thread
    # 参数两个，一个是需要运行的函数名，第二个是函数的参数，作为元组使用，可为空
    # 注意：如果函数只有一个参数，参数后需要有一个逗号
    thread.start_new_thread(loop1, ())
    thread.start_new_thread(loop2, ())
    print("All done at:", time.ctime())
if __name__ == '__main__':
    main()
    while True:
        time.sleep(1)

# 03
import time
import _thread as thread
def loop1(in1):
    print("Start loop 1 at :",time.ctime())
    print("我是参数", in1)
    time.sleep(4)
    print("End loop 1 at: ", time.ctime())
def loop2(in1, in2):
    print("Start loop 1 at :",time.ctime())
    print("我是参数", in1, "和参数", in2)
    time.sleep(2)
    print("End loop 2 at: ", time.ctime())
def main():
    print("Starting at: ", time.ctime())
    thread.start_new_thread(loop1, ("wangda", ))
    thread.start_new_thread(loop2, ("linge", "doeoe", ))
    print("All done at:", time.ctime())
if __name__ == '__main__':
    main()
    while True:
        time.sleep(1)

• threading：通行的包
  • 直接利用threading.thread生成Thread实例
    1、t = threadind.Thread(target=xxx, args=(xxx, ))
    2、t.start():启动多线程
    3、t.join():等待多线程执行完成
  • 案例04，05

# 04
import time
import threading
def loop1(in1):
    print("Start loop 1 at :",time.ctime())
    print("我是", in1)
    time.sleep(4)
    print("End loop 1 at: ", time.ctime())
def loop2(in1, in2):
    print("Start loop 2 at :",time.ctime())
    print("你是", in1, "他是", in2)
    time.sleep(2)
    print("End loop 2 at: ", time.ctime())
def main():
    print("Starting at: ", time.ctime())
    t1 = threading.Thread(target=loop1, args=("吴邪",))
    t1.start()
    t2 = threading.Thread(target=loop2, args=("王胖子","黑瞎子"))
    t2.start()
    print("All done at:", time.ctime())
if __name__ == '__main__':
    main()
    while True:
        time.sleep(10)

# 05
import time
import threading
def loop1(in1):
    print("Start loop 1 at :",time.ctime())
    print("我是", in1)
    time.sleep(4)
    print("End loop 1 at: ", time.ctime())
def loop2(in1, in2):
    print("Start loop 2 at :",time.ctime())
    print("你是", in1, "他是", in2)
    time.sleep(2)
    print("End loop 2 at: ", time.ctime())
def main():
    print("Starting at: ", time.ctime())
    t1 = threading.Thread(target=loop1, args=("吴邪",))
    t1.start()
    t2 = threading.Thread(target=loop2, args=("王胖子","黑瞎子"))
    t2.start()
    t1.join()
    t2.join()
    print("All done at:", time.ctime())
if __name__ == '__main__':
    main()
    while True:
        time.sleep(10)

• 守护线程daemon
  • 如果在程序中将子线程设置成守护线程，则子线程会在主线程借宿的时候自动退出
  • 一般认为，守护线程不重要或者不允许离开主线程独立运行
  • 守护线程案例能否有效跟环境相关
• 案例06，07

# 06
import time
import threading
def loop1():
    print("Start loop 1 at :")
    print("我是:")
    time.sleep(2)
    print("End loop 1 at: ")
print("Starting at: ")
t1 = threading.Thread(target=loop1, args=())
t1.start()
print("All done at:")
time.sleep(1)

# 07
import time
import threading
def loop1():
    print("Start loop 1 at :")
    print("我是:")
    time.sleep(2)
    print("End loop 1 at: ")
print("Starting at: ")
t1 = threading.Thread(target=loop1, args=())
t1.setDaemon(True)
# t1.daemon = (True)
t1.start()
print("All done at:")
time.sleep(1)

• 线程常用属性
  • threading.currentThread：返回当前线程变量
  • threading.enumerate：返回一个包含正在运行的线程的list，
  • threading.activeCount：返回正在运行的线程数量，跟Len(threading.ennumrate)
  • threading.setName：给线程设置名称
  • threading.getName: 得到线程的名称
• 案例08

# 08
import time
import threading
def loop1():
    print("Start loop 1 at :",time.ctime())
    time.sleep(4)
    print("End loop 1 at: ", time.ctime())
def loop2():
    print("Start loop 2 at :",time.ctime())
    time.sleep(2)
    print("End loop 2 at: ", time.ctime())
def loop3():
    print("Start loop 3 at :",time.ctime())
    time.sleep(5)
    print("End loop 3 at: ", time.ctime())
def main():
    print("Starting at: ", time.ctime())
    t1 = threading.Thread(target=loop1, args=())
    t1.setName("THR1")
    t1.start()
    t2 = threading.Thread(target=loop2, args=())
    t2.setName("THR2")
    t2.start()
    t3 = threading.Thread(target=loop3, args=())
    t3.setName("THR3")
    t3.start()
    time.sleep(3)
    for thr in threading.enumerate():
        print("Processing is {0}".format(thr.getName()))
    print("Processing Num is {0}:".format(threading.activeCount()))
    print("All done at:", time.ctime())
if __name__ == '__main__':
    main()

• 用继承方式，直接继承为threading.Thread的子类
  • 继承Thread
  • 重写run函数
• 案例09

# 09
import threading
import time
# 定义一个类，继承自threading.Thread
class MyThread(threading.Thread):
    def __init__(self, arg):
        super(MyThread, self).__init__()
        self.arg = arg
    # 必须重新run函数，run函数是真正要执行的内容
    def run(self):
        time.sleep(2)
        print("the arg for this class is {0}".format(self.arg))
for i in range(5):
    t = MyThread(i)
    t.start()
    t.join()
print("Main thread is done")

• 案例10：一个标准的工业风案例

# 10
import threading
from time import ctime, sleep
loop = [4,2]
class ThreadFunc:
    def __init__(self, name):
        self.name = name

    # 相当run函数，执行功能
    def loop(self, nloop, nsec):
        """
        :param nloop: loop函数的名称
        :param nsec: 系统休眠时间
        :return:
        """
        print('Start loop', nloop, 'at', ctime())
        sleep(nsec)
        print('Done loop', nloop, 'at', ctime())

def main():
    print("Starting at: ", ctime())
    # ThreadFunc("loop").loop 等同以下两个表达式
    t = ThreadFunc("loop")
    # t.loop
    t1 = threading.Thread(target=t.loop, args=("LOOP1", 4))
    t2 = threading.Thread(target=ThreadFunc('loop').loop, args=("LOOP2", 2))
    t1.start()
    t2.start()
    t1.join()
    t2.join()
    print("All done at: ", ctime())

if __name__ == '__main__':
    main()

• 共享变量
  • 共享变量：当多个线程同时访问一个变量的时候，
• 案例11

# 11
import threading
sum = 0
loopsum = 100000
# 定义一个Lock的实例
lock = threading.Lock()

def add():
    global sum, loopsum
    for i in range(1, loopsum):
        # 上锁
        lock.acquire()
        sum += 1
        # 开锁
        lock.release()

def min():
    global sum, loopsum
    for i in range(1, loopsum):
        # 上锁
        lock.acquire()
        sum -= 1
        # 开锁
        lock.release()
# 第一次没有定义LOCK实例时，变量sum由于共享使用问题，出现不一致的结果（具体参考“原子操作”）
# 第二次定义 LOCK 实例，通过限定变量使用，使多线程时，同一时间只能由一个线程在占用
if __name__ == '__main__':
    print("Starting.....{0}".format(sum))
    t1 = threading.Thread(target=add, args=())
    t2 = threading.Thread(target=min, args=())
    t1.start()
    t2.start()
    t1.join()
    t2.join()
    print("Done.....{0}".format(sum))

• 解决方式：锁，信号
• 锁（LOCK）
• 线程安全问题
• 如果一个资源/变量，对于多线程来说，不用加锁也不会引起任何问题，则成为线程安全
• 线程不安全变量类型：list, set, dict
• 线程安全变量类型： queue
• 死锁问题：当两个线程各自占有锁，又交替申请对方锁而造成的无法继续执行的困局
• 解决方式：等待时间设置 案例13

# 13
import threading
import time

lock1 = threading.Lock()
lock2 = threading.Lock()

def func_1():
    print("func_1 starting......")
    lock1.acquire(timeout=4)
    print("func_1 acquired lock1.....")
    time.sleep(2)
    print("func_1 waiting for lock2....")
    if lock2.acquire(timeout=2):
        print("func_1 had has lock2....")
        lock2.release()
        print("func_1 released lock2....")
    else:
        print("func_1 had not acquired lock2...")
    lock1.release()
    print("func_1 released lock1...")
    print("func_1 done.....")

def func_2():
    print("func_2 starting.....")
    lock2.acquire(timeout=2)
    print("func_2 acquired lock2....")
    time.sleep(4)
    print("func_2 waiting for lock1...")
    lock1.acquire(timeout=6)
    print("func_2 acquired lock1....")
    lock1.release()
    print("func_2 released lock1....")
    lock2.release()
    print("func_2 released lock2....")
    print("func_2 done")

if __name__ == '__main__':
    print("Main program starting.......")
    t1 = threading.Thread(target=func_1, args = ())
    t2 = threading.Thread(target=func_2, args = ())
    t1.start()
    t2.start()
    t1.join()
    t2.join()
    print("Main program done.....")

• semphore：允许一个资源最多由几个多线程同时使用 案例14
• Timer：利用多线程，在指定时间后启动一个功能 案例15

# 14
import threading
import time

# 参数定义最多几个线程同时使用资源
semphore = threading.Semaphore(5)
def func():
    if semphore.acquire():
        for i in range(5):
            print(threading.currentThread().getName() + "get semphore")
        time.sleep(15)
        semphore.release()
        print(threading.currentThread().getName() + "release semphore")

for i in range(8):
    t1 = threading.Thread(target=func)
    t1.start()

# 15
import threading
import time

def func():
    print("I am running.....")
    time.sleep(4)
    print("T am done........")
if __name__ == '__main__':
    t = threading.Timer(6, func)
    t.start()
    i = 0
    while True:
        print("{0}".format(i))
        time.sleep(3)
        i += 1

• 可重入锁
  • 一个锁，可以倍一个线程多次申请
  • 主要解决递归调用的时候，需要申请锁的情况
  • threading.RLock() 案例16

# 16
import threading
import time
class MyThread(threading.Thread):
    def run(self):
        global num
        time.sleep(1)
        if mutex.acquire(3):
            num += 1
            msg = self.name + 'set num to ' + str(num)
            print(msg)
            mutex.acquire()
            mutex.release()
            mutex.release()

num = 0
mutex = threading.RLock()
def test1():
    for i in range(5):
        t = MyThread()
        t.start()

if __name__ == '__main__':
    test1()

• 线程替代方案
• subprocess
  • 完全跳过线程，使用进程
  • 是Python进程的主要替代方案
  • python2.4后引入
• multiprocessing
  • 使用threading接口派生，使用子进程
  • 允许为多核或者多cpu派生进程
• concurrent.futures
  • 新的异步执行模块
  • 任务级别的操作
  • Python3.2后引入

3、重点研究下多进程

• 进程间通讯（InterprocessCommunication, IPC）
• 进程之间无任何共享状态
• 进程的创建
  • 直接生成Process实例对象， 案例1
  • 派生子类 案例2
• 在OS中查看pid,ppid以及他们的关系 案例3

# 1
import multiprocessing
from time import sleep, ctime
def clock(interval):
    while True:
        print("The time is %s" % ctime())  # 子进程
        sleep(interval)
if __name__ == '__main__':
    p = multiprocessing.Process(target=clock, args=(5,))
    p.start()
    while True:
        print("Main process sleeping......")  # 主进程
        sleep(1)

# 2
import multiprocessing
from time import sleep, ctime

class ClockProcess(multiprocessing.Process):
    def __init__(self, interval):
        super().__init__()
        self.interval = interval
    def run(self):
        while True:
            print("The time is %s" % ctime())  # 子进程
            sleep(self.interval)

if __name__ == '__main__':
    p = ClockProcess(3)
    p.start()
    while True:
        print("Main process sleeping......")  # 主进程
        sleep(1)

# 3
from multiprocessing import Process
import os
def info(title):
    print(title)
    print("module name: ", __name__)
    # 得到父类进程的ID
    print("parent process: ", os.getppid())
    # 得到子类进程的ID
    print(" subprocess: ", os.getpid())
def f(name):
    info("funcion f")
    print("hello", name)
if __name__ == '__main__':
    info("main line")
    p = Process(target = f, args=("bobb", ))
    p.start()
    p.join()

生产者消费者模型

• 生产者消费者问题
  • 一个模型：可以用来搭建消息队列
  • queue是一个用来存放变量的数据结构，先进先出，内部元素排序
  • 案例consumer_producer_01

# consumer_producer_01
import threading
import time
import queue  # Python3

class Producer(threading.Thread):
    def run(self):
        global q
        count = 0
        while True:
            if q.qsize() < 1000:
                for i in range(100):
                    count += 1
                    msg = "生成产品" + str(count)
                    # put是从queue中存入一个值
                    q.put(msg)
                    print(msg)
            time.sleep(0.5)

class Consumer(threading.Thread):
    def run(self):
        global q
        while True:
            if q.qsize() > 200:
                for i in range(5):
                    # get是从queue中取出一个值
                    msg = self.name + "消费了" + q.get()
                    print(msg)
            time.sleep(1)

if __name__ == '__main__':
    q = queue.Queue()
    for i in range (500):
        q.put("产品" + str(i))
    for i in range(2):
        Producer().start()
    for i in range(5):
        Consumer().start()

• JoinableQueue([maxsize])
  • 允许项目的使用者通知生成者项目已经被成功处理
  • task_done(): 使用者使用此方法发出信号，表示q.get()的返回项目已经被处理
  • join(): 生产者调用此方法进行阻塞，直到队列中所有的项目均被处理
  • 案例consumer_producer_02

# consumer_producer_02
import multiprocessing
from time import ctime

def consumer(input_q):
    print("Into consumer:", ctime())
    while True:
        # 处理项
        item = input_q.get()
        print("Pull", item, "out of q")
        input_q.task_done()
    print("Out of consumer:", ctime())

def producer(sequence, output_q):
    print("Into producer:", ctime())
    for item in sequeuce:
        output_q.put(item)
        print("Put", item, "into q")
    print("Out of producer:", ctime())

if __name__ == '__main__':
    q = multiprocessing.JoinableQueue()
    cons_p = multiprocessing.Process(target=consumer, args=(q, ))
    cons_p.daemon = True
    cons_p.start()

    sequence = [1,2,3,4,5,6]
    producer(sequence, q)
    q.join()

• 在Queue中加入“哨兵”的使用
  • 案例consumer_producer_03

# consumer_producer_03
import multiprocessing
from time import ctime

def consumer(input_q):
    print("Into consumer:", ctime())
    while True:
        # 处理项
        item = input_q.get()
        if item is None:
            break
        print("Pull", item, "out of q")
       # input_q.task_done()
    print("Out of consumer:", ctime())

def producer(sequence, output_q):
    print("Into producer:", ctime())
    for item in sequeuce:
        output_q.put(item)
        print("Put", item, "into q")
    print("Out of producer:", ctime())

if __name__ == '__main__':
    q = multiprocessing.Queue()
    cons_p = multiprocessing.Process(target=consumer, args=(q, ))
    # cons_p.daemon = True
    cons_p.start()

    sequence = [1,2,3,4,5,6]
    producer(sequence, q)
    q.put(None)  # 在执行完生产者所有动作之后，放入“哨兵”值
    cons_p.join()

4、总结，以及困惑

线程，进程这一部分是自学习以来遇到比较难啃的知识点。难啃的点再于，还没有很顺利理清楚代码的逻辑和流程，以及应该应用在何处上。
当然，这也是急不来的，就像一个月前对于字典类型迟迟未能通关，只到最近在用Python获取MySQL数据库表信息的时刻，才通彻。