【Python篇】scrapy爬虫

前言

Scrapy是一个为了爬取网站或API数据，提取结构性数据而编写的应用框架。 可以应用在包括数据挖掘，信息处理或存储历史数据等一系列的程序中。

---

该文章是通过scrapy爬取https://www.yelp.com/关于[Restaurants]模块每个餐馆的菜单图片」

---

1、工具安装

1.1 安装Python环境
Mac一般自带 Python 2.7，不用额外安装

1.2 安装pip

sudo install pip

1.3 安装pycharm
https://www.jetbrains.com/pycharm/

2、创建项目

scrapy startproject imageSpider

3、开始爬虫

3.1 设置Item

class MenuImageItem(scrapy.Item):
   # define the fields for your item here like:
   image_urls = scrapy.Field()
   image_name = scrapy.Field()

3.2 设置pipelines

class TestspiderPipeline(ImagesPipeline):
    default_headers = {

    }

    def get_media_requests(self, item, info):
        for image_url in item['image_urls']:
            yield Request(image_url, meta={'item': item, 'index': item['image_urls'].index(image_url)})

    def file_path(self, request, response=None, info=None):
        print '---------------'
        item = request.meta['item']  # 通过上面的meta传递过来item
        index = request.meta['index']  # 通过上面的index传递过来列表中当前下载图片的下标
        # 图片文件名，item['carname'][index]得到汽车名称，request.url.split('/')[-1].split('.')[-1]得到图片后缀jpg,png
        image_guid = request.url.split('/')[-1]
        # os.path.splitext()[0]
        # 图片下载目录 此处item['country']即需要前面item['country']=''.join()......,否则目录名会变成\u97e9\u56fd\u6c7d\u8f66\u6807\u5fd7\xxx.jpg
        img_file_name = u'source/{0}/images/{1}'.format(item['image_name'],image_guid)

        return img_file_name

3.3 设置setting

ITEM_PIPELINES = {
    # 'imageDemo.pipelines.ImagedemoPipeline': 300,
    # 'imageDemo.pipelines.MenuImagePipeline': 300,
    'imageDemo.pipelines.SubDoubanImgDownloadPipeline': 300,
}
IMAGES_STORE = '/Users/chenxiao/Desktop/'

3.4 编写spider

import scrapy
from scrapy.selector import Selector
from imageDemo.items import ImagedemoItem
from imageDemo.items import MenuImageItem

import sys
reload(sys)
sys.setdefaultencoding('utf-8')

class imageSpider(scrapy.Spider):
    name = "imageSpider"
    allowed_domains = ["yelp.com"]
    start_urls = [
        # "https://www.yelp.com/search?cflt=restaurants&find_loc=San%20Francisco%2C%20CA&start=0",
        # "https://www.yelp.com/biz_photos/peoples-bistro-san-francisco-4?tab=menu",
        "https://www.yelp.com/biz_photos/peter-luger-brooklyn-2?tab=menu",
    ]

    def parse(self, response):
        sel = Selector(response)
        menuList = sel.xpath('//*[@id="super-container"]/div[2]/div/div[2]/div[2]/ul/li/div/img/@src').extract()
        print menuList
        # print '>>>>>>>'
        yield MenuImageItem(image_urls=menuList, image_name='peter-luger')

        # list = sel.xpath('//*[@id="wrap"]/div[3]/div[2]/div[2]/div/div[1]/div[1]/div/ul/li/div/div/div/div/div[2]/div[1]/div[1]/div[1]/div[1]/h3/a[@class="lemon--a__373c0__IEZFH link__373c0__29943 link-color--blue-dark__373c0__1mhJo link-size--inherit__373c0__2JXk5"]/text()').extract()
        # print list
        # for item in list:
        #     print item
        #     print '>>>>>'
            # yield ImagedemoItem(name=item)

4、下载资源

scrapy crawl imageSpider

5、项目源码

https://github.com/GeeksChen/ScrapyDemo

注意：

1、command not found: scrapy
https://blog.csdn.net/EDDYCJY/article/details/77482228
ln -s /Users/xinyinhe/Library/Python/2.7/bin/scrapy /usr/local/bin/scrapy
找到python/bin/scrapy 然后进行软链接