什么是进程(process)？
An executing instance of a program is called a process.
Each process provides the resources needed to execute a program. A process has a virtual address space, executable code, open handles to system objects, a security context, a unique process identifier, environment variables, a priority class, minimum and maximum working set sizes, and at least one thread of execution. Each process is started with a single thread, often called the primary thread, but can create additional threads from any of its threads.
程序并不能单独运行，只有将程序装载到内存中，系统为它分配资源才能运行，而这种执行的程序就称之为进程。程序和进程的区别就在于：程序是指令的集合，它是进程运行的静态描述文本；进程是程序的一次执行活动，属于动态概念。
在多道编程中，我们允许多个程序同时加载到内存中，在操作系统的调度下，可以实现并发地执行。这是这样的设计，大大提高了CPU的利用率。进程的出现让每个用户感觉到自己独享CPU，因此，进程就是为了在CPU上实现多道编程而提出的。
有了进程为什么还要线程？
进程有很多优点，它提供了多道编程，让我们感觉我们每个人都拥有自己的CPU和其他资源，可以提高计算机的利用率。很多人就不理解了，既然进程这么优秀，为什么还要线程呢？其实，仔细观察就会发现进程还是有很多缺陷的，主要体现在两点上：
-
进程只能在一个时间干一件事，如果想同时干两件事或多件事，进程就无能为力了。
-
进程在执行的过程中如果阻塞，例如等待输入，整个进程就会挂起，即使进程中有些工作不依赖于输入的数据，也将无法执行。
例如，我们在使用qq聊天， qq做为一个独立进程如果同一时间只能干一件事，那他如何实现在同一时刻 即能监听键盘输入、又能监听其它人给你发的消息、同时还能把别人发的消息显示在屏幕上呢？你会说，操作系统不是有分时么？但我的亲，分时是指在不同进程间的分时呀， 即操作系统处理一会你的qq任务，又切换到word文档任务上了，每个cpu时间片分给你的qq程序时，你的qq还是只能同时干一件事呀。
再直白一点， 一个操作系统就像是一个工厂，工厂里面有很多个生产车间，不同的车间生产不同的产品，每个车间就相当于一个进程，且你的工厂又穷，供电不足，同一时间只能给一个车间供电，为了能让所有车间都能同时生产，你的工厂的电工只能给不同的车间分时供电，但是轮到你的qq车间时，发现只有一个干活的工人，结果生产效率极低，为了解决这个问题，应该怎么办呢？。。。。没错，你肯定想到了，就是多加几个工人，让几个人工人并行工作，这每个工人，就是线程！
什么是线程(thread)？
线程是操作系统能够进行运算调度的最小单位。它被包含在进程之中，是进程中的实际运作单位。一条线程指的是进程中一个单一顺序的控制流，一个进程中可以并发多个线程，每条线程并行执行不同的任务
A thread is an execution context, which is all the information a CPU needs to execute a stream of instructions.
Suppose you're reading a book, and you want to take a break right now, but you want to be able to come back and resume reading from the exact point where you stopped. One way to achieve that is by jotting down the page number, line number, and word number. So your execution context for reading a book is these 3 numbers.
If you have a roommate, and she's using the same technique, she can take the book while you're not using it, and resume reading from where she stopped. Then you can take it back, and resume it from where you were.
Threads work in the same way. A CPU is giving you the illusion that it's doing multiple computations at the same time. It does that by spending a bit of time on each computation. It can do that because it has an execution context for each computation. Just like you can share a book with your friend, many tasks can share a CPU.
On a more technical level, an execution context (therefore a thread) consists of the values of the CPU's registers.
Last: threads are different from processes. A thread is a context of execution, while a process is a bunch of resources associated with a computation. A process can have one or many threads.
Clarification: the resources associated with a process include memory pages (all the threads in a process have the same view of the memory), file descriptors (e.g., open sockets), and security credentials (e.g., the ID of the user who started the process).
进程与线程的区别？
- Threads share the address space of the process that created it; processes have their own address space.
- Threads have direct access to the data segment of its process; processes have their own copy of the data segment of the parent process.
- Threads can directly communicate with other threads of its process; processes must use interprocess communication to communicate with sibling processes.
- New threads are easily created; new processes require duplication of the parent process.
- Threads can exercise considerable control over threads of the same process; processes can only exercise control over child processes.
- Changes to the main thread (cancellation, priority change, etc.) may affect the behavior of the other threads of the process; changes to the parent process does not affect child processes.
Python threading模块
线程有2种调用方式，如下：
直接调用
import threading
import time
def sayhi(num): #定义每个线程要运行的函数
print("running on number:%s" %num)
time.sleep(3)
if __name__ == '__main__':
t1 = threading.Thread(target=sayhi,args=(1,)) #生成一个线程实例
t2 = threading.Thread(target=sayhi,args=(2,)) #生成另一个线程实例
t1.start() #启动线程
t2.start() #启动另一个线程
print(t1.getName()) #获取线程名
print(t2.getName())输出：
running on number:1
running on number:2
Thread-1
Thread-2继承式调用：
import threading
import time
class MyThread(threading.Thread):
def __init__(self,num):
super(MyThread,self).__init__()
self.num = num
def run(self):#定义每个线程要运行的函数
print("running on number:%s" %self.num)
time.sleep(3)
if __name__ == '__main__':
t1 = MyThread(1)
t2 = MyThread(2)
t1.start()
t2.start()输出：
running on number:1
running on number:2守护线程
Some threads do background tasks, like sending keepalive packets, or performing periodic garbage collection, or whatever. These are only useful when the main program is running, and it's okay to kill them off once the other, non-daemon, threads have exited.
Without daemon threads, you'd have to keep track of them, and tell them to exit, before your program can completely quit. By setting them as daemon threads, you can let them run and forget about them, and when your program quits, any daemon threads are killed automatically.
import time
import threading
def run(n):
print('[%s]------running----\n' % n)
time.sleep(1)
print('--done--')
def main():
for i in range(5):
t = threading.Thread(target=run,args=[i,])
t.start()
t.join(1)
print('starting thread', t.getName())
m = threading.Thread(target=main,args=[])
m.setDaemon(True) #将main线程设置为Daemon线程,它做为程序主线程的守护线程,当主线程退出时,m线程也会退出,由m启动的其它子线程会同时退出,不管是否执行完任务
m.start()
m.join(timeout=2)
print("---main thread done----")Note：Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event
.
.线程锁(互斥锁Mutex)
一个进程下可以启动多个线程，多个线程共享父进程的内存空间，也就意味着每个线程可以访问同一份数据，此时，如果2个线程同时要修改同一份数据，会出现什么状况？
import time
import threading
def addNum():
global num #在每个线程中都获取这个全局变量
print('--get num:',num )
time.sleep(1)
num -=1 #对此公共变量进行-1操作
num = 100 #设定一个共享变量
thread_list = []
for i in range(100):
t = threading.Thread(target=addNum)
t.start()
thread_list.append(t)
for t in thread_list: #等待所有线程执行完毕
t.join()
print('final num:', num )输出：
--get num: 100
--get num: 100
--get num: 100
...
--get num: 100
--get num: 100
--get num: 100
--get num: 100
--get num: 100
final num: 0正常来讲，这个num结果应该是0， 但在python 2.7上多运行几次，会发现，最后打印出来的num结果不总是0，为什么每次运行的结果不一样呢？ 哈，很简单，假设你有A,B两个线程，此时都 要对num 进行减1操作， 由于2个线程是并发同时运行的，所以2个线程很有可能同时拿走了num=100这个初始变量交给cpu去运算，当A线程去处完的结果是99，但此时B线程运算完的结果也是99，两个线程同时CPU运算的结果再赋值给num变量后，结果就都是99。那怎么办呢？ 很简单，每个线程在要修改公共数据时，为了避免自己在还没改完的时候别人也来修改此数据，可以给这个数据加一把锁， 这样其它线程想修改此数据时就必须等待你修改完毕并把锁释放掉后才能再访问此数据。
*注：不要在3.x上运行，不知为什么，3.x上的结果总是正确的，可能是自动加了锁，我在2.7中运行这个程序和在3.x上效果一样
加锁版本
import time
import threading
def addNum():
global num #在每个线程中都获取这个全局变量
print('--get num:',num )
time.sleep(1)
lock.acquire() #修改数据前加锁
num -=1 #对此公共变量进行-1操作
lock.release() #修改后释放
num = 100 #设定一个共享变量
thread_list = []
lock = threading.Lock() #生成全局锁
for i in range(100):
t = threading.Thread(target=addNum)
t.start()
thread_list.append(t)
for t in thread_list: #等待所有线程执行完毕
t.join()
print('final num:', num )输出：
--get num: 100
--get num: 100
......
--get num: 100
final num: 0RLock（递归锁）
说白了就是在一个大锁中还要再包含子锁
输入：
import threading,time
def run1():
print("grab the first part data")
lock.acquire()
global num
num +=1
lock.release()
return num
def run2():
print("grab the second part data")
lock.acquire()
global num2
num2+=1
lock.release()
return num2
def run3():
lock.acquire()
res = run1()
print('--------between run1 and run2-----')
res2 = run2()
lock.release()
print(res,res2)
if __name__ == '__main__':
num,num2 = 0,0
lock = threading.RLock()
for i in range(10):
t = threading.Thread(target=run3)
t.start()
while threading.active_count() != 1: #等于1代表所有线程结束了
print(threading.active_count())
else:
print('----all threads done---')
print(num,num2)输出： 每次运行的输出结果都不一样
grab the first part data
--------between run1 and run2-----
grab the second part data
1 1
grab the first part data
--------between run1 and run2-----
grab the second part data
2 2
grab the first part data
--------between run1 and run2-----
grab the second part data
3 3
grab the first part data
--------between run1 and run2-----
grab the second part data
4 4
grab the first part data
--------between run1 and run2-----
grab the second part data
5 5
grab the first part data
--------between run1 and run2-----
grab the second part data
6 6
grab the first part data
--------between run1 and run2-----
grab the second part data
7 7
grab the first part data
--------between run1 and run2-----
4
grab the second part data
8 8
grab the first part data
3
--------between run1 and run2-----
3
grab the second part data
3
9 9
3
grab the first part data
2
--------between run1 and run2-----
2
grab the second part data
2
10 10
2
----all threads done---
10 10
信号量 semaphore
互斥锁 同时只允许一个线程更改数据，而Semaphore是同时允许一定数量的线程更改数据 ，比如厕所有3个坑，那最多只允许3个人上厕所，后面的人只能等里面有人出来了才能再进去。
import threading,time
def run(n):
semaphore.acquire()
time.sleep(1)
print("run the thread: %s\n" %n)
semaphore.release()
if __name__ == '__main__':
semaphore = threading.BoundedSemaphore(5) #最多允许5个线程同时运行
for i in range(23):
t = threading.Thread(target=run,args=(i,))
t.start()
while threading.active_count() != 1:
pass #print threading.active_count()
else:
print('----all threads done---')输出：
run the thread: 1
run the thread: 4
run the thread: 2
run the thread: 0
run the thread: 3
run the thread: 5
run the thread: 8
run the thread: 7
run the thread: 6
run the thread: 9
run the thread: 10
run the thread: 13
run the thread: 11
run the thread: 14
run the thread: 12
run the thread: 15
run the thread: 16
run the thread: 18
run the thread: 19
run the thread: 17
run the thread: 20
run the thread: 21
run the thread: 22
----all threads done---
Timer
This class represents an action that should be run only after a certain amount of time has passed
Timers are started, as with threads, by calling their start() method. The timer can be stopped (before its action has begun) by calling thecancel() method. The interval the timer will wait before executing its action may not be exactly the same as the interval specified by the user.
def hello():
print("hello, world")
t = Timer(30.0, hello)
t.start() # after 30 seconds, "hello, world" will be printedEvents
An event is a simple synchronization object;
the event represents an internal flag, and threads
can wait for the flag to be set, or set or clear the flag themselves.
event = threading.Event()
# a client thread can wait for the flag to be set
event.wait()
# a server thread can set or reset it
event.set()
event.clear()
If the flag is set, the wait method doesn’t do anything.
If the flag is cleared, wait will block until it becomes set again.
Any number of threads may wait for the same event.
通过Event来实现两个或多个线程间的交互，下面是一个红绿灯的例子，即起动一个线程做交通指挥灯，生成几个线程做车辆，车辆行驶按红灯停，绿灯行的规则
import threading,time
import random
def light():
if not event.isSet():
event.set() #wait就不阻塞 #绿灯状态
count = 0
while True:
if count < 10:
print('--green light on---')
elif count <13:
print('--yellow light on')
elif count <20:
if event.isSet():
event.clear()
print('--red light on---')
else:
count = 0
event.set() #打开绿灯
time.sleep(1)
count +=1
def car(n):
while 1:
time.sleep(random.randrange(10))
if event.isSet(): #绿灯
print("car [%s] is running.." % n)
else:
print("car [%s] is waiting for the red light.." %n)
if __name__ == '__main__':
event = threading.Event()
Light = threading.Thread(target=light)
Light.start()
for i in range(3):
t = threading.Thread(target=car,args=(i,))
t.start()
queue队列
queue is especially useful in threaded programming when information must be exchanged safely between multiple threads.
class queue.Queue(maxsize=0) #先入先出
class queue.LifoQueue(maxsize=0) #last in fisrt out
class queue.PriorityQueue(maxsize=0) #存储数据时可设置优先级的队列
Constructor for a priority queue. maxsize is an integer that sets the upperbound limit on the number of items that can be placed in the queue. Insertion will block once this size has been reached, until queue items are consumed. If maxsize is less than or equal to zero, the queue size is infinite.
The lowest valued entries are retrieved first (the lowest valued entry is the one returned by sorted(list(entries))[0]). A typical pattern for entries is a tuple in the form: (priority_number, data).
exception queue.Empty
Exception raised when non-blocking get() (or get_nowait()) is called on a Queue object which is empty.
exception queue.Full
Exception raised when non-blocking put() (or put_nowait()) is called on a Queue object which is full.
Queue.qsize()
Queue.empty() #return True if empty
Queue.full() # return True if full
Queue.put(item, block=True, timeout=None)
Put item into the queue. If optional args block is true and timeout is None (the default), block if necessary until a free slot is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Full exception if no free slot was available within that time. Otherwise (block is false), put an item on the queue if a free slot is immediately available, else raise the Full exception (timeout is ignored in that case).
Queue.put_nowait(item)
Equivalent to put(item, False).
Queue.get(block=True, timeout=None)
Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).
Queue.get_nowait()
Equivalent to get(False).
Two methods are offered to support tracking whether enqueued tasks have been fully processed by daemon consumer threads.
Queue.task_done()
Indicate that a formerly enqueued task is complete. Used by queue consumer threads. For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.
If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).
Raises a ValueError if called more times than there were items placed in the queue.
Queue.join() block直到queue被消费完毕
生产者消费者模型
在并发编程中使用生产者和消费者模式能够解决绝大多数并发问题。该模式通过平衡生产线程和消费线程的工作能力来提高程序的整体处理数据的速度。
为什么要使用生产者和消费者模式
在线程世界里，生产者就是生产数据的线程，消费者就是消费数据的线程。在多线程开发当中，如果生产者处理速度很快，而消费者处理速度很慢，那么生产者就必须等待消费者处理完，才能继续生产数据。同样的道理，如果消费者的处理能力大于生产者，那么消费者就必须等待生产者。为了解决这个问题于是引入了生产者和消费者模式。
什么是生产者消费者模式
生产者消费者模式是通过一个容器来解决生产者和消费者的强耦合问题。生产者和消费者彼此之间不直接通讯，而通过阻塞队列来进行通讯，所以生产者生产完数据之后不用等待消费者处理，直接扔给阻塞队列，消费者不找生产者要数据，而是直接从阻塞队列里取，阻塞队列就相当于一个缓冲区，平衡了生产者和消费者的处理能力。
下面来学习一个最基本的生产者消费者模型的例子：两个生产者 两个消费者
import threading
import queue
import time
def producer(name):
count = 1
while True:
q.put("%s 包子 %s" %(name,count))
print("%s生产了包子，总共生产了%s包子" %(name,count))
count += 1
time.sleep(1)
def consumer(name):
while True:
print("%s 取到 %s" %(name,q.get()))
time.sleep(1)
q = queue.Queue()
p = threading.Thread(target=producer,args=('****',))
p1 = threading.Thread(target=producer,args=('#####',))
c = threading.Thread(target=consumer,args=('hhhhh',))
c1 = threading.Thread(target=consumer,args=('lllll',))
p.start()
p1.start()
c.start()
c1.start()输出：
****生产了包子，总共生产了1包子
#####生产了包子，总共生产了1包子
hhhhh 取到 **** 包子 1
lllll 取到 ##### 包子 1
****生产了包子，总共生产了2包子
#####生产了包子，总共生产了2包子
hhhhh 取到 **** 包子 2
lllll 取到 ##### 包子 2
****生产了包子，总共生产了3包子
#####生产了包子，总共生产了3包子
lllll 取到 **** 包子 3
hhhhh 取到 ##### 包子 3
****生产了包子，总共生产了4包子
#####生产了包子，总共生产了4包子
lllll 取到 **** 包子 4
hhhhh 取到 ##### 包子 4
#####生产了包子，总共生产了5包子
****生产了包子，总共生产了5包子
hhhhh 取到 ##### 包子 5
lllll 取到 **** 包子 5
****生产了包子，总共生产了6包子
hhhhh 取到 **** 包子 6
#####生产了包子，总共生产了6包子
lllll 取到 ##### 包子 6多进程：
在from multiprocessing import Process中，Process会有红色波浪线，但是可以使用
from multiprocessing import Process
import time
def f(name):
time.sleep(1)
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('hcl',))
p.start()
p.join()
查看进程id号：
输入：
from multiprocessing import Process
import time
import os
def info(title):
print(title)
print('module name:',__name__)
print('parent process:',os.getppid())
print('process id:',os.getpid())
print('\n')
def f(name):
info('function')
print('hello',name)
if __name__ == '__main__':
info('main process line')
p = Process(target=f,args=('hcl',))
p.start()
p.join()输出：
main process line
module name: __main__
parent process: 37996
process id: 4268
function
module name: __mp_main__
parent process: 4268
process id: 23728
hello hcl进程间通信
不同进程间内存是不共享的，要想实现两个进程间的数据交换，可以用以下方法：
Queue
from multiprocessing import Process,Queue
def f(q):
q.put([42,None,'hello'])
if __name__ == '__main__':
q = Queue()
p = Process(target = f, args = (q,))
p.start()
print(q.get())
p.join(输出：
[42, None, 'hello']Pipes
from multiprocessing import Process,Pipe
def f(conn):
conn.send([42,None,'hello'])
conn.close()
if __name__ == '__main__':
parent_conn,child_conn = Pipe()
p = Process(target = f,args = (child_conn,))
p.start()
print(parent_conn.recv())
p.join()输出
[42, None, 'hello']
Managers
from multiprocessing import Process,Manager
def f(d,l):
d[1] = '1'
d['2'] = 2
d[0.25] = None
l.append(1)
print(l)
if __name__ == '__main__':
with Manager() as manager:
d = manager.dict()
l = manager.list(range(5))
p_list = []
for i in range(10):
p = Process(target = f,args = (d,l))
p.start()
p_list.append(p)
for res in p_list:
res.join()
print(d)
print(l)输出：
[0, 1, 2, 3, 4, 1]
[0, 1, 2, 3, 4, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1, 1, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1, 1, 1, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1, 1, 1, 1, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1, 1, 1, 1, 1, 1, 1]
[0, 1, 2, 3, 4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
{1: '1', '2': 2, 0.25: None}
[0, 1, 2, 3, 4, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]进程同步：
from multiprocessing import Process,Lock
def f(l,i):
l.acquire()
try:
print('hello world',i)
finally:
l.release()
if __name__ == '__main__':
lock = Lock()
for num in range(10):
Process(target = f,args = (lock,num)).start()输出：
hello world 2
hello world 0
hello world 3
hello world 1
hello world 5
hello world 8
hello world 4
hello world 6
hello world 9
hello world 7进程池
from multiprocessing import Process,Pool
import time
def Foo(i):
time.sleep(2)
return i+100
def Bar(arg):
print('-->exec done:',arg)
if __name__ == '__main__':
pool = Pool(3)
for i in range(10):
pool.apply_async(func = Foo,args=(i,),callback = Bar)
#pool.apply(func = Foo,args = (i,))
print('end')
pool.close()
pool.join()输出：
end
-->exec done: 100
-->exec done: 102
-->exec done: 101
-->exec done: 103
-->exec done: 104
-->exec done: 105
-->exec done: 106
-->exec done: 107
-->exec done: 108
-->exec done: 109
六、有兴趣接电子设计相关小型项目的请加下群，每个项目一般在1000元以内，非诚勿扰
扫一扫
1344
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
