aiohttp 多任务异步协程式爬虫爬取某站图片

跳跳没虎

已于 2022-02-03 16:48:59 修改

阅读量870

点赞数 4

分类专栏：笔记文章标签： python pycharm 开发语言爬虫

于 2021-11-11 18:27:09 首次发布

本文链接：https://blog.csdn.net/weixin_45059947/article/details/121273569

版权

笔记专栏收录该内容

5 篇文章 0 订阅

订阅专栏

这是一个使用Python的异步请求库aiohttp从指定网站下载高清图片的脚本。用户可以输入页数和文件存储路径，脚本将爬取图片原始地址并保存为jpg格式，同时处理文件名中可能存在的非法字符。注意，由于版权原因，实际网址并未提供。

摘要由CSDN通过智能技术生成

爬取的是图片原地址并不是预览图
点进图片后的大高清图
#版权原因不能发图
代码如下
只需改变文件储存路径即可

import requests
import os
import asyncio
import aiohttp


async def getpic(oneof_datalist):
    async with aiohttp.ClientSession() as session:
        if 'coverImage' in oneof_datalist['data']:
            url = 'https://img2.huashi6.com/'+oneof_datalist['data']['coverImage']['originalPath']
            filename = oneof_datalist['data']['title']
            sets = ['/', '\\', ':', '*', '?', '"', '<', '>', '|']
            for char in filename:
                if char in sets:
                    filename = filename.replace(char, '')
            filename.split()
            async with await session.get(url) as response:
                picdata = await response.read()
                cpicPath = picPath + '/' + filename + '.jpg'
                with open(cpicPath, 'wb+') as f:
                    f.write(picdata)
                    print(filename, '下载成功')
        elif 'worksList'in oneof_datalist['data']:
            for i in oneof_datalist['data']['worksList']:
                url = 'https://img2.huashi6.com/'+i['coverImage']['originalPath']
                filename = i['title']
                sets = ['/', '\\', ':', '*', '?', '"', '<', '>', '|']
                for char in filename:
                    if char in sets:
                        filename = filename.replace(char, '')
                filename.split()
                async with await session.get(url) as response:
                    picdata = await response.read()
                    cpicPath = picPath + '/' + filename + '.jpg'
                    with open(cpicPath, 'wb+') as f:
                        f.write(picdata)
                        print(filename, '下载成功')

headers ={
    'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'
    }

page = int(input('enter  pages'))
picPath = r'{}'.format(input('plz input a  filepath '))

if not os.path.exists(picPath):
     os.makedirs(picPath)

json_list = []
for page in range(1,page):
    url = 'https://rt.huashi6.com/front/index/load_pc_data?_ts_=1636363387274&cursor={}-1636361232855'.format(page)
    response_data = requests.post(url = url,headers = headers).json()
    json_list.append(response_data)

datalist = []
for i in json_list:
    datalist.extend(i['data']['datas'])
print(len(datalist))

tasks = []
for one in datalist:
    c = getpic(one)
    task = asyncio.ensure_future(c)
    tasks.append(task)

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))