看文档学爬虫(11)——异步加载

最新推荐文章于 2024-10-30 19:27:31 发布

sinat_18131557

最新推荐文章于 2024-10-30 19:27:31 发布

阅读量146

点赞数

分类专栏：爬虫 Python 文章标签： python 爬虫

Python 同时被 2 个专栏收录

66 篇文章 40 订阅

订阅专栏

爬虫

12 篇文章 0 订阅

订阅专栏

转自：http://f61be319.wiz03.com/share/s/3S6-cp1BIQ952yXKyj02PIM43QcqFo3DzAbT2Y6fvo3IV3SQ

什么是异步加载

异步加载就是在浏览网页时候，通过继续加载一部分内容，而非后续全部内容的方式。

示例

import requests
import re
from bs4 import BeautifulSoup
headers = {
    'accept': 'text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36'
}
urls = ['https://www.pexels.com/?page={}'.format(str(i)) for i in range(1, 2)]
photos = []
for url in urls:
    wb_data = requests.get(url, headers=headers)
    soup = BeautifulSoup(wb_data.text, 'lxml')
    imgs = soup.select('article.photo-item > a > img')
    for img in imgs:
        photo = img.get('src')
        photos.append(photo)
path = 'E://yibu/'
for item in photos:
    data = requests.get(item, headers=headers)
    photo_name = re.findall('\d+\/(.*?)\?h', item)
    print(photo_name)
    if photo_name:
        fp = open(path + photo_name[0], 'wb')
        fp.write(data.content)
        fp.close()