这个项目我们的主题是爬腾讯视频的电影信息，包括电影名和描述
1 创建工程
打开一个终端，在想要的位置使用以下命令创建一个新的工程文件夹，名为TXspider
D:\VSCode\scrapy爬虫>scrapy startproject TXspider
cd Txspider
2 新建一个爬虫
新建的时候一定要先给它一个url。
D:\VSCode\scrapy爬虫\TXspider>scrapy genspider txsr v.qq.com
这里我们需要爬取的完整页面网址是https://v.qq.com/channel/cartoon?listpage=1&channel=cartoon&iarea=1
3 修改setting.py
- 修改USER_AGENT=‘Mozilla、5.0’（简单理解就是将你的爬虫程序伪装成一个浏览器请求）
- 修改ROBOTSTXT_OBEY=False（即不遵循机器人协议，否则很多东西爬取不了）
- 修改DOWNLOAD_DELAY=1（修改下载延迟，不要太慢也不能太快）
ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 1
DEFAULT_REQUEST_HEADERS = {
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'en',
'User-Agent':'Mozilla/5.0'
}
4 确认要提取的数据项
item.py定义要提取的内容。
这里我们准备爬取名字和介绍以及图片的url。
import scrapy
class TxspiderItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
name = scrapy.Field()
description=scrapy.Field()
image_url=scrapy.Field()
5 写爬虫程序（txsr.py)。
parse()方法负责处理response并返回处理的数据以及(/或)跟进的URL。
该方法及其他的Request回调函数必须返回一个包含 Request 及(或) Item 的可迭代的对象。
import scrapy
#引入TxspiderItem
from ..items import TxspiderItem
class TxsrSpider(scrapy.Spider):
name = 'txsr'
allowed_domains = ['v.qq.com']
start_urls = ['https://v.qq.com/x/bu/pagesheet/list?append=1&channel=cartoon&iarea=1&listpage=2&offset=0&pagesize=30']
offset=0
def parse(self, response):
#定义一个容器保存爬取的信息
items=TxSpiderItem()
#这部分是爬取部分，使用xpath的方式选择信息
lists=response.xpath('//div[@class="list_item"]')
for i in lists:
items['name']=i.xpath('./a/@title').get()
items['description']=i.xpath('./div/div/@title').get()
items['image_url']=i.xpath('./a/img/@src').get()
yield items
#爬取了150项的内容，通过分析HTML页面可以发现每30个一页。
if self.offset<120:
self.offset+=30
url='https://v.qq.com/x/bu/pagesheet/list?append=1&channel=cartoon&iarea=1&listpage=2&offset={}&pagesize=30'.format(
str(self.offset))
yield scrapy.Request(url=url,callback=self.parse)
此刻，在终端中可以运行一下爬虫，看看是否写的有错，
D:\VSCode\scrapy爬虫\TXspider>scrapy crawl txsr
没有错误的话会显示如下信息：
上面信息表示，我们已经获取了信息，接下来我们开始进行信息的储存。
使用Pipeline处理数据
当Item在Spider中被收集之后，它将会被传递到Pipeline，一些组件会按照一定的顺序执行对Item的处理。
Pipeline经常进行一下一些操作：
清理HTML数据
验证爬取的数据(检查item包含某些字段)
查重(并丢弃)
将爬取结果保存到数据库中
这里只进行简单的将数据储存在json文件的操作。pipeline有三种方法：process_item，open_spider，close_spider。我这里后两种都没有用到，因为我让json文件一直处于打开没有关闭。
#引入文件
from scrapy.exceptions import DropItem
import json
class TxspiderPipeline(object):
def __init__(self):
#打开文件
self.file = open('data.json', 'w', encoding='utf-8')
#该方法用于处理数据
def process_item(self, item, spider):
#读取item中的数据
line = json.dumps(dict(item), ensure_ascii=False) + "\n"
#写入文件
self.file.write(line)
#返回item
return item
#该方法在spider被开启时被调用。
def open_spider(self, spider):
pass
#该方法在spider被关闭时被调用。
def close_spider(self, spider):
pass
要使用pipeline,首先需要注册，即在setting.py配置文件中添加。如果没有改类名的话，setting.py中自动生成的有，只需要去掉注释。
ITEM_PIPELINES = {
'TXspider.pipelines.TxspiderPipeline': 300,
}
会在工程的目录下生成一个data.json文件，如图
{"name": "斗罗大陆", "description": "此生不悔入唐门", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/m441e3rjq9kwpsc1607693898908/220"}
{"name": "开心锤锤", "description": "普通锤锤的爆笑日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200dwnknik1605525478040/220"}
{"name": "武神主宰", "description": "武神跌落，浴火少年再起", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/7q544xyrava3vxf1610952810187/220"}
{"name": "灵剑尊", "description": "天地三界，我为至尊！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/2w2legt0g8z26al1592380967544/220"}
{"name": "猪屁登", "description": "和猪屁登一起传递正能量", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200ydcnajl1611567824235/220"}
{"name": "狐妖小红娘", "description": "万物有界，爱恨无由", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/0sdnyl7h86atoyt1606023864287/220"}
{"name": "绝世武魂", "description": "吞噬龙血，少年逆天崛起", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200no18bci1596439143337/220"}
{"name": "万界仙踪", "description": "仙魔一念，人间千载", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/5cuf8ahvxvm25871600070585676/220"}
{"name": "雪鹰领主", "description": "超凡归来 薪火相传", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/sifd2an7kx2h9h81606379317134/220"}
{"name": "伍六七之玄武国篇", "description": "最强首席，开启奇险之旅！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200y9i58mv1611116313905/220"}
{"name": "妖神记", "description": "踏足武道巅峰", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/yl6lapwmmx5ivew1584947181675/220"}
{"name": "天荒战神", "description": "问天一剑，能否天荒地老", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200tw780l11592843745261/220"}
{"name": "独步逍遥", "description": "少年热血闯红尘", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200qqsk3cv1590046238923/220"}
{"name": "飞狗MOCO", "description": "柯基与主人的搞笑日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mn3pui7xiqdm8wr1568190677/220"}
{"name": "吞噬星空", "description": "人类的进化，才是唯一答案", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/324olz7ilvo2j5f1606560252337/220"}
{"name": "无上神帝", "description": "仙王觉醒，重归万界巅峰", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200ilydv1a1589510037678/220"}
{"name": "武庚纪", "description": "神力觉醒，三界大战！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/ipmc5u3dwb48mv21568708922/220"}
{"name": "万界神主", "description": "陨落古神，遨游苍蓝", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/y0jueuihog64xhb1578989707/220"}
{"name": "我是大神仙", "description": "用现代知识称霸古代仙界", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002009u3yfca1569763477/220"}
{"name": "迷你小洞", "description": "迷你世界爆笑同人动画", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200fb5nu5c1577170162/220"}
{"name": "一念永恒", "description": "还有这样沙雕的修仙番？", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/ww18u675tfmhas61597169528078/220"}
{"name": "星辰变", "description": "穿星辰沧海 赴羽立之约", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/0s8n49g3g1rv1oz1587613647738/220"}
{"name": "一人之下 第3季", "description": "豆瓣高分人气动画再度回归", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mtxrcjfijj77gze1571818196/220"}
{"name": "墓王之王", "description": "武林风云，波澜再起！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/72g4vwb42f558pu1572417139/220"}
{"name": "万圣街", "description": "西洋神怪的东方合租日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/awnia0n2erqryf31602557824776/220"}
{"name": "动漫弹弹乐", "description": "一周鹅友神弹幕TOP10", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200afm3ueb1573116226/220"}
{"name": "万界法神", "description": "美好世界由我拯救！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002007995z4v1608390825912/220"}
{"name": "小品一家人", "description": "小品的搞笑温馨日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200pk5azl41574674152/220"}
{"name": "仙风剑雨录", "description": "人妖殊途，情丝不断", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200hc38s5x1597151605357/220"}
{"name": "非人哉", "description": "神仙妖怪的爆笑日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/wk85h1a859a83501611741186944/220"}
{"name": "书灵记", "description": "吟诗读句！以唐诗作战", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200b1plp0l1565595882/220"}
{"name": "元气食堂", "description": "吃货熊大卫的美食日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002005sty8ad1591957136473/220"}
{"name": "迷你世界奥特曼动画", "description": "迷你大陆的奥特曼", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200txrxgnm1588825716031/220"}
{"name": "画江湖之轨夜行", "description": "最强赛车手的诞生", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/tfc35o92twfe3vi1604562417483/220"}
{"name": "面膜妈妈养娃-小视频特别版", "description": "暖心家庭的温馨生活", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002008ksgll91594310025528/220"}
{"name": "小狮子赛几 - 小视频特别版", "description": "赛几不寻常的日常故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc0020082smllf1576571655/220"}
{"name": "星骸骑士", "description": "银河冒险，血战星骸！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/umpnsyqfu7f60se1606706432400/220"}
{"name": "魔道祖师", "description": "羡云篇霸气回归", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/k4mutekomtrdbux1565158963/220"}
{"name": "元气食堂-小视频特别版", "description": "吃货熊大卫的美食日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200v4ju2qd1583839679890/220"}
{"name": "王者脑洞所", "description": "王者英雄的爆笑故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200abcdfyg1584432023476/220"}
{"name": "飞狗MOCO之宠物店的小秘密", "description": "萌宠天团日常逗趣", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200idkryt51571293542/220"}
{"name": "迷你兄弟", "description": "在游戏里生存的爆笑故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200by5u21r1591182071815/220"}
{"name": "伍六七之最强发型师", "description": "豆瓣高分国漫爆笑大作", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200v31r9ez1570900982/220"}
{"name": "小品一家人 四川方言版", "description": "幽默搞笑的一家四口", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002003973ymu1592477280699/220"}
{"name": "熊小兜", "description": "萌娃与父母的日常对话", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/sbyk5j06t0zeowb1557906494/220"}
{"name": "尊宝爸爸搞笑动漫", "description": "超人气搞笑动画", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200g1m7twe1584608113748/220"}
{"name": "吃鸡爆笑兄弟", "description": "游戏中吃鸡的爆笑小故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200aiqqj3j1591181826906/220"}
{"name": "长安幻街", "description": "一街两界、昼夜交替", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200cn6dafh1609235795059/220"}
{"name": "nana娜娜酱", "description": "玛丽苏少女喵历险记", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200vpp9eny1577193855/220"}
{"name": "魁拔之殊途", "description": "万众期待，魁拔归来", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/eh4fr6jrni2gty81604892106273/220"}
{"name": "铁头与橘子-小视频特别版", "description": "霸道总裁与傻白甜", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002008velhyo1586927805926/220"}
{"name": "熊卷&妈妈", "description": "熊卷与妈妈的爆笑日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200imcgz631588824344483/220"}
{"name": "武动乾坤", "description": "少年出山动天下，武者入世定乾坤", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/jg1skog3zvrv9ur1593312414145/220"}
{"name": "鼠星星", "description": "鼠星星的吃货哲学", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200ru7sma71592988627305/220"}
{"name": "迷你变形金刚", "description": "迷你战士挑战外族入侵", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002006gi0zrh1592825151019/220"}
{"name": "飞狗MOCO之宠物店的小秘密-小视频特别版", "description": "萌宠天团日常逗趣", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200jrhza3s1585035689544/220"}
{"name": "迷你吃鸡小队", "description": "迷你世界同人吃鸡动画", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002008qaklcv1590550460554/220"}
{"name": "奶瓶小星", "description": "搞怪小屁孩的逗趣生活", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200cnsrk4j1572533189/220"}
{"name": "魔道祖师Q", "description": "温馨治愈的可爱日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200fdthd811597984316967/220"}
{"name": "阿巳与小铃铛", "description": "吃不完的狗粮都在这里", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200bznlvdt1604556229963/220"}
{"name": "天行九歌", "description": "超高颜值的权谋史诗", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/rm3tmmat4li8uul1554835621/220"}
{"name": "我的狐仙女帝", "description": "狐仙女帝穿越寻夫复仇", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200yow6jot1592193129023/220"}
{"name": "小品一家人-小视频特别版", "description": "小品爆笑日常小视频版", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200lufafia1574925471/220"}
{"name": "我是不白吃", "description": null, "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200rsd0rse1606821265550/220"}
{"name": "铅笔动画", "description": "铅笔人也能让你笑起来", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002005dpfjl91574926594/220"}
{"name": "叫我僵小鱼 日常篇 第二季", "description": "僵小鱼的欢脱暖心之旅", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/4wq0y3lws5njpot1557943327/220"}
{"name": "阿衰", "description": "神级漫画家的史诗剧作", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/06xnwuxhcfrwj1c1597402116872/220"}
{"name": "刺客伍六七 第1季", "description": "神秘美少女的遗愿清单", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/9jh7p3r7ljkxkmc1526440483/220"}
{"name": "动态漫画·斗破苍穹", "description": "好嗨呀！少年激燃战群豪！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002001hfdcja1606286954574/220"}
{"name": "MC战纪", "description": "传说人物him的故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200h7hdwas1594307175959/220"}
{"name": "小冷哥", "description": "小冷哥的都市生活糗事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200v95dyl61587970612235/220"}
{"name": "水豚汤馆", "description": "有事没事泡个汤", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002003a3020z1594985508456/220"}
{"name": "胡闹鬼阿月-小视频特别版", "description": "阿月和小伙伴们的搞笑日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200123yo7j1592904414813/220"}
{"name": "狗哥杰克苏", "description": "先虐后撩甜宠霸总动画", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200b2j866v1576828537/220"}
{"name": "斗破苍穹 第3季", "description": "异火臣服 萧炎掌握佛怒火莲", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/o0ytzgvq6o08e9o1562393177/220"}
{"name": "超能力小苏", "description": "隐藏富豪兼超能力拥有者", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200q54broy1584609168034/220"}
{"name": "动态漫画·斗罗大陆2 绝世唐门 荣耀篇", "description": "唐三之后，荣耀再临", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/hvaed7hei6jdskw1543463932/220"}
{"name": "孙娇Jiao", "description": "霸气娇姐的大学生活", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200kgwotjl1590393288727/220"}
{"name": "迷你世界大冒险", "description": "一起在迷你世界冒险吧~", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200v1dlqty1595732862313/220"}
{"name": "九夜神传", "description": "天赋少女冒险异界大陆", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200t2nedm41606967286883/220"}
{"name": "非人哉精选集", "description": "精选非人哉爆笑日常段子", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/1paeg9n4o8vvpsp1523510263/220"}
{"name": "猪小屁生活篇", "description": "猪小屁与大美的日常生活", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200vvf9d171569579964/220"}
{"name": "西行纪 普通话版", "description": "热血西行，披荆斩棘", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/0gsf9fytppje54d1569567552/220"}
{"name": "我的狐仙女帝-小视频特别版", "description": "狐仙女帝穿越寻夫复仇", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002009qhcc271590124414457/220"}
{"name": "都市妖怪生存指南-小视频特别版", "description": "都市妖怪治愈日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200g9q7p061592903519976/220"}
{"name": "一人之下 第2季", "description": "风起云涌罗天大醮", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/3enwc74hj562xjd1508914052/220"}
{"name": "杰克大魔王 - 小视频特别版", "description": "杰克大魔王来到现实世界", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200x1hj7n91573035736/220"}
{"name": "首席御灵师", "description": "神州浩土，万物有灵", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200siw6d9n1590061974379/220"}
{"name": "超能力小苏 第2季", "description": "超能力拥有者搞笑再袭", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200d3hh4o41595491066886/220"}
{"name": "狼狼少女蓝-小视频特别版", "description": "校园魔幻爱情故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200hmhwwsb1596713724047/220"}
{"name": "万界春秋", "description": "口吞烈阳，燃铸我身！", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200sgy2t2x1588142967924/220"}
{"name": "狗哥杰克苏-小视频特别版", "description": "先虐后撩甜宠霸总特别版", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200fezciqv1576830661/220"}
{"name": "全职高手 第2季", "description": "荣耀不灭，王者卷土重来", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/3r7bnv3gdykfdok1600163123669/220"}
{"name": "王者爆笑动画", "description": "《王者荣耀》同人动画", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200u9yx7d61594915394402/220"}
{"name": "达夫玩游戏", "description": "达夫和游戏好友爆笑日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200pd17wk21575943872/220"}
{"name": "狼人沈天", "description": "科幻系狼人变成爱豆", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200rvf9e081584105612203/220"}
{"name": "萌芽熊", "description": "熊童子的搞笑治愈课", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002005935gbi1586436871072/220"}
{"name": "八爪鱼手绘定格动画", "description": "动物们的定格日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002002xqt6a91572533937/220"}
{"name": "茶啊二中爆笑小视频", "description": "茶啊二中的爆笑校园生活", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200w6q00va1573805610/220"}
{"name": "草帽肥肥", "description": "肥肥的日常爆笑神转折", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200ar85jcg1583308582717/220"}
{"name": "斗破苍穹 第2季", "description": "萧炎归来 风云再起", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/8j37ljw0f8hy6ha1514885218/220"}
{"name": "末世觉醒之溯源", "description": "科幻悬疑原创3D巨制", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200opm367n1608600965265/220"}
{"name": "魔王老公欠调教", "description": "总裁变身成狼谈恋爱", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200u5exrba1590160212136/220"}
{"name": "是个鸡爪", "description": "正能量逗趣光头少女", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002002g8ez451586321041232/220"}
{"name": "全职法师 第4季", "description": "莫凡觉醒恶魔系能力", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/vc5b1spq46fqfgn1562404485/220"}
{"name": "由你玩四年：阿银和他的吃鸡队友", "description": "在海岛战场上的宿舍战队", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200oweyle51577787393/220"}
{"name": "动态漫画·斗破苍穹之大主宰 第2季", "description": "追求着那主宰之路", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200wryhe9x1597047687921/220"}
{"name": "超能力小苏-小视频特别版", "description": "隐藏富豪兼超能力拥有者", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200ws8wt7h1584330345443/220"}
{"name": "动态漫画·绝世武神 第2季", "description": "废柴逆袭，武道至尊！", "image_url": "//puui.qpic.cn/tv/0/1224328579/0"}
{"name": "狼人沈天 - 小视频特别版", "description": "狼人化成人形出道做爱豆", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200wf6tde01584104554776/220"}
{"name": "一人之下 第1季", "description": "异人不老少女", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/0vjw392pdevdan3t1467257477.jpg/220"}
{"name": "通灵妃", "description": "年度鬼畜爱情大戏", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/jg2a5feze5bryj21544668734/220"}
{"name": "暗界神使 普通话版", "description": "双男主联手探离奇诡案", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200mqgogho1611906637088/220"}
{"name": "穿书自救指南", "description": "书穿反派开启洗白之路", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/e13cyu7h55k62ps1599367133268/220"}
{"name": "虎墩小镖师", "description": "小男孩虎墩的成长之旅", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200kkswxf11584439165766/220"}
{"name": "动态漫画·琅寰书院", "description": "仙师出手，一拳致胜", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002005dvpvip1568624719/220"}
{"name": "雄兵连", "description": "世纪大战一触即发", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/p0pcfbdk318ry3m1525318295/220"}
{"name": "坦克吃鸡大作战 第二季", "description": "刺激的坦克激斗", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200dbfjods1586256159219/220"}
{"name": "动态漫画·斗罗大陆3龙王传说 第1季", "description": "斗罗大陆，神龙降临", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200apsjzeq1575948511/220"}
{"name": "我的超能力女友", "description": "超能少女与富二代宅男", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200ss4k5hl1591182448473/220"}
{"name": "英雄再临", "description": "最强英雄退休再就业", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/q72jd29a3oxflsr1562400511/220"}
{"name": "动态漫画·斗罗大陆2 绝世唐门", "description": "唐三之后，斗罗大陆新星降临。", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/atk62w0bq58dnq91515566753/220"}
{"name": "皮蛋的铅笔世界", "description": "手绘温暖的二次元小故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200g6kmdlf1584081478468/220"}
{"name": "宠宠联萌", "description": "宠宠联萌的爱宠日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200r8zuukp1590157206396/220"}
{"name": "动态漫画·完美世界 第1季", "description": "大荒天帝，独断万古", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200adehour1597388826199/220"}
{"name": "异世界中药铺", "description": "破除阻碍，异界振兴中药", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200dtt7zfd1571312892/220"}
{"name": "英雄觉醒", "description": "超级英雄正在全面苏醒", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200vbd3je91611199809200/220"}
{"name": "雄兵连之诸天降临", "description": "抗击外星入侵", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/ei44lqqq0fsg5aq1547432177/220"}
{"name": "迷你世界木鱼狗铛求生记", "description": "木鱼大冒险搞笑之旅", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc0020067mfoyh1598319867900/220"}
{"name": "一禅小和尚", "description": "和尚的开悟人生", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/ij4yjidgfnw346d1585046146569/220"}
{"name": "斗破苍穹 第1季", "description": "一季完结 少年乘空翱翔", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/xvlhj7cglyfwx8x1481886522/220"}
{"name": "择天记", "description": "龙凤反目长生大破谜局", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002000he7xqq1576836196/220"}
{"name": "画江湖之不良人 第1季", "description": "群雄侠隐仗剑", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/o85sk7hqhbw3g9nt1468470635.jpg/220"}
{"name": "侠岚", "description": "玄幻武侠动画", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/y96malg2m60eznpt1444942378.jpg/220"}
{"name": "坦克吃鸡大作战 第一季", "description": "刺激的坦克激斗", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200xdxnk6a1589367213222/220"}
{"name": "全职法师 第3季", "description": "法师集结，决战狂魔", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/8vdu7i5hr7anq5q1535824602/220"}
{"name": "迷你小镇", "description": "爆笑成长故事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc0020036paydj1596771795575/220"}
{"name": "豆豆猪日记", "description": "可爱豆豆猪萌你没商量", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc002005v7rju31569398321/220"}
{"name": "蘑菇头搞笑日常", "description": "蘑菇头F4搞笑生活日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc0020087wbh7w1586327538316/220"}
{"name": "动态漫画·至尊神魔", "description": "废柴少年的逆袭成最强炼丹师", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200v1r31dd1567479351/220"}
{"name": "罗小黑战记", "description": "可爱小黑猫", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/ygci7rbfq3celp8t1444944536.jpg/220"}
{"name": "植物大战僵尸吃鸡大战", "description": "植物和僵尸也吃鸡", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200o9slp1i1578899256/220"}
{"name": "快把我哥带走 普通话版", "description": "暴力妹和二货哥", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/1lcp2pf46ms2r3o1597655541079/220"}
{"name": "外卖仙人成天", "description": "一个修仙世界的外卖员", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc0020038f6jxa1598167623478/220"}
{"name": "鼠星星 第2季", "description": "鼠星星和娜娜子的趣事", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200jl37z131606451710697/220"}
{"name": "侠客阿坤的日常", "description": "宫本的爆笑游戏日常", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200cc4o11l1587739494429/220"}
{"name": "绝命响应", "description": "集合力量对抗最强智能人", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/nj8sf17z6w3bgak1559181246/220"}
{"name": "猫妖的诱惑 正式版", "description": "自古爱恨两难，当然是插他两刀", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/z4ulitk78335qjq1550115240/220"}
{"name": "有声漫画·贺少的闪婚暖妻", "description": "聪明可爱的萌宝", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/mzc00200aho73sm1592968489520/220"}
{"name": "全职高手 第1季", "description": "十年热血书写电竞辉煌", "image_url": "//puui.qpic.cn/vcover_vt_pic/0/ttqxo9s3cblgoflt1471595407.jpg/220"}
下载图片
在上文中我们只获得了图片的url并没有下载，这里我们进行图片下载的编写。
首先，需要在Items.py文件中添加存取图片的路径。
完整的Items.py代码如下：
import scrapy
class TxspiderItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
name = scrapy.Field()
description=scrapy.Field()
image_url=scrapy.Field()
#图片地址
image_path=scrapy.Field()
关于Pipeline可以新建一个文件（这样的话可以和之前上面的那个的区分，也可以控制它们的优先级）
新建一个ImgPipelines.py
代码如下：
import scrapy
from scrapy.pipelines.images import ImagesPipeline
from scrapy.exceptions import DropItem
class ImgPipeline(ImagesPipeline):
#通过抓取的图片url获取一个Request用于下载
def get_media_requests(self,item,info):
#返回Request根据图片图片url下载
yield scrapy.Request(item['image_url'])
#当下载请求完成后执行该方法
def item_completed(self, results, item, info):
#获取下载地址
image_path = [x['path'] for ok, x in results if ok]
#判断是否成功
if not image_path:
raise DropItem("Item contains no images")
#将地址存入item
item['image_path'] = image_path
return item
然后再setting.py中注册一下，并且添加IMAGES_URLS_FIELD和IMAGES_STORE。
ITEM_PIPELINES = {
'TXspider.pipelines.TxspiderPipeline': 300,
'TXspider.ImgPipelines.ImgPipeline': 100,#这里100是它的优先级
}
IMAGES_URLS_FIELD ="image_url" #对应item里面设定的字段，取到图片的url
IMAGES_STORE='D:/VScode/myimg' #这里是我保存图片的地址，可以随意。
然后运行一下，在过程中我遇到了一个问题：
报错：ValueError: Missing scheme in request url:
我们很容易知道是因为没有找到正确的url。这里通过在网上搜索了一番，，，（让我明白了一个道理，还是自己动脑比较好。）到这一步了，肯定是下载图片的url不对，想想确实有点不对，通过上面的输出就知道了
然后，修改一下爬虫程序即我的txsr.py
reurl="http:"+i.xpath('./a/img/@src').get()
items['image_url']=reurl
然后，再次运行，大功告成！
成功爬取到了图片。
参考链接：
https://blog.csdn.net/ck784101777/article/details/104468780/
https://blog.csdn.net/zjiang1994/article/details/52779537
扫一扫
232
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
