异步爬虫的实现

无限制目的：在爬虫中使用异步去提高，爬虫的性能

异步爬虫的方式：

一：多进程，多线程：

        好处：当阻塞发起时，我们可以单独开启线程或者进程，阻塞操作就可以异步实现

        弊端：不可以无限制的开启进程或者线程

二：进程池，线程池：

        使用类：from multiprocessing.dummy import Pool

        好处：可以降低进程或者线程创建或销毁的频率，从而很好的减少系统的损失

        弊端：池中进程或线程数量有限

        比较：

单线程耗时：

线程池耗时：

但确实肉眼可见其存在紊乱现象

下面是使用其处理梨视频的代码

import requests
import re
from multiprocessing import Pool
head = {
    "user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36",
    "referer":"https://www.pearvideo.com/category_4",
    "cookie":"__secdyid=cca1d79ad157836c3acf36de0919e939df5b5add140ad9a6021642930997; PEAR_UUID=ce058fb4-48aa-4a36-9782-597651dd14e3; _uab_collina=164293098384419249496175; UM_distinctid=17e865159dc281-03384e7b10c10d-f791b31-144000-17e865159dd3d3; p_h5_u=4A84EE5E-4E4F-4D5C-8E6C-A765816418A3; acw_tc=2f61f27b16463758878505365e288ded8e2ebb74cb9374baf97bf05776bf00; JSESSIONID=32A4B9EC6CF2683E4716DBE754813201; Hm_lvt_9707bc8d5f6bba210e7218b8496f076a=1645782956,1646375889; CNZZDATA1260553744=420920985-1642920725-https%3A%2F%2Fwww.baidu.com%2F|1646373255; Hm_lpvt_9707bc8d5f6bba210e7218b8496f076a=1646376280; SERVERID=a7cc60ddba048546c9441d2558c201d4|1646376843|1646375887"
}
def geturl():
    urllist = []
    for i in range(3):
        url = "https://www.pearvideo.com/category_loading.jsp?reqType=5&categoryId=4&start="
        text = requests.get(url,headers=head).text
        code = re.findall('<a href="video_(.*?)" class="vervideo-lilink actplay">',text)
        for i in code:
            urllist.append(i)
    return urllist
def getvideo(url):
    referer = "https://www.pearvideo.com/video_" + url
    newUrl = "https://www.pearvideo.com/videoStatus.jsp?contId=" + url
    newhead = {
           "user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36",
            "referer":referer
    }
    text = requests.get(newUrl,headers=newhead).text
    texturl = re.findall('"srcUrl":"(.*?)"',text)[0]
    textlist = str(texturl).split("/")
    number = re.findall('(.*?)-',textlist[-1])[0]
    texturl = texturl.replace(number,"cont-"+url)
    content = requests.get(texturl,headers=head).content
    with open("梨视频"+url+".mp4","wb") as op:
        op.write(content)
    print("视频" + url +"保存成功")
if __name__ == "__main__":
    urllist = geturl()
    pool = Pool(10)
    pool.map(getvideo,urllist)