Scrapy框架整合英雄缩略图（APP）

1.自定义需求
英雄缩略图，手机查看太麻烦，一个个翻，看起来不够直观，想要实现按英雄文件夹汇总，分类保存在本地，英雄数据信息保存在数据库中！
2.需求分析
a)根据英雄名称创建文件夹
b)根据英雄所有皮肤图片链接保存图片
c)英雄的属性信息保存MongoDb
3.实现思路
a)技术栈：Fiddler数据分析+Python+Scrapy框架+MongoDb数据库+virtualenv沙箱
b)抓包分析
①首先设置Fiddler代理，手机连接无线网，设置无线网代理
②拿到英雄属性数据
http://gpcd.gtimg.cn/upload/qqtalk/lol_hero/1d/hero_list.js
③英雄皮肤图片数据
http://lol.qq.com/biz/hero/skins.js
④数据格式

⑤英雄英文名称en_name匹配图片关键字，ok，数据关系已经找到了

⑥代码实现，发送链接，解析数据，保存

Spider：请求数据，保存图片

# -*- coding: utf-8 -*-
import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule,Request
import json,re,os


class SkinSpider(CrawlSpider):
    name = 'Skin'
    allowed_domains = ['lol.qq.com']
    start_urls = ['http://gpcd.gtimg.cn/upload/qqtalk/lol_hero/1d/hero_list.js']
    count = 0
    rules = (
        #Rule(LinkExtractor(allow=r'lol_hero/'), callback='parse_item', follow=True),
    )

    def parse_start_url(self, response):
        datas = json.loads(response.text)
        dicnames = {}
        meta = {}
        for data in datas:
            dicnames[data["en_name"]] = data["name"]
            yield data
        meta["datanames"] = dicnames
        yield Request('http://lol.qq.com/biz/hero/skins.js',meta = meta,callback=self.parse_skin,dont_filter=True)

    def parse_skin(self,respomse):
        strskins = re.findall(r'{"Aatrox.*.}]}',respomse.text)[0]
        datas = json.loads(strskins)
        datanames = respomse.meta["datanames"]
        for key in datas:
            if datas[key]:
                for item in datas[key]:
                    meta  = {}
                    meta["dir"] = datanames[key]
                    meta["filename"] = item["name"]
                    yield Request("http://ossweb-img.qq.com/images/lol/appskin/{}.jpg".format(item["id"]),meta = meta, callback=self.parse_image, dont_filter=True)


    def parse_image(self,response):
        self.create_dir(response.meta)
        dirpath = response.meta["dir"].strip().rstrip("\\")
        filename = response.meta["filename"].strip().rstrip("\\")
        fullname = "./SkinsPic/{}/{}.jpg".format(dirpath,filename)
        with open(fullname, 'wb') as f:
            f.write(response.body)
            self.count = self.count+1
            print('success count:',self.count)

    def create_dir(self,meta):
        basepath = './SkinsPic'
        dirpath = basepath+'/'+meta["dir"].strip().rstrip("\\")
        if not os.path.exists(basepath):
            os.makedirs(basepath)
        if not os.path.exists(dirpath):
            os.makedirs(dirpath)

Piplines代码:英雄属性数据插入到MongoingDb

# -*- coding: utf-8 -*-
from pymongo import MongoClient
import datetime

class LolskinPipeline(object):

    def __init__(self):
        client = MongoClient('mongodb://localhost:27017')
        db = client['LOL']
        self.collection = db["skin"]
        self.count = 0

    def process_item(self, item, spider):
        print('********************************************')
        print(item,item.keys())
        if 'en_name' in item.keys():
            item["time"] =datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
            self.collection.insert_one(item)
            self.count = self.count + 1
            return self.count
        else:
            return 'no item'

settings配置

ITEM_PIPELINES = {
   'LOlSkin.pipelines.LolskinPipeline': 300,
}

4.效果图


5.总结
①理解Scrapy框架
②熟悉Linux开发环境
③熟练virtualenv