scrapy框架入门与使用

1.scrapy的安装

python -m pip install --upgrade pip
 
pip install wheel
 
pip install lxml
 
pip install twisted
 
pip install pywin32
 
pip install scrapy

2.scrapy的入门案例

1.创建项目

打开cmd，输入(默认是在C:\Users\Administrator>这个目录下，你可以自行切换）：

scrapy startproject myfirstPj
 
cd my firstPj
 
scrapy genspider baidu www.baidu.com

2.修改setting

修改三项内容，第一个是不遵循机器人协议，第二个是下载间隙，由于下面的程序要下载多个页面，所以需要给一个间隙（不给也可以，只是很容易被侦测到），第三个是请求头，添加一个User-Agent，第四个是打开一个管道

ROBOTSTXT_OBEY=False
DOWNLOAD_DELAY=1DEFAULT_REQUEST_HEADERS={'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language':'en','User-Agent':'Mozilla/5.0(WindowsNT6.2;WOW64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/27.0.1453.94Safari/537.36'}ITEM_PIPELINES={'TXmovies.pipelines.TxmoviesPipeline':300,}

3.确认要提取的数据，item项

item定义你要提取的内容（定义数据结构），比如我提取的内容为电影名和电影描述，我就创建两个变量。Field方法实际上的做法是创建一个字典，给字典添加一个建，暂时不赋值，等待提取数据后再赋值。下面item的结构可以表示为：{'name':'','descripition':''}。

#-*-coding:utf-8-*-
 
#Defineherethemodelsforyourscrapeditems
 
#
 
#Seedocumentationin:
 
#https://docs.scrapy.org/en/latest/topics/items.html
 
importscrapy
 
classTxmoviesItem(scrapy.Item):
 
        #definethefieldsforyouritemherelike:
 
        #name=scrapy.Field()
 
        name=scrapy.Field()
 
        description=scrapy.Field()

4.写爬虫程序

items['name']=i.xpath('./a/@title')[0]
items['name']=i.xpath('./a/@title').extract()
items['name']=i.xpath('./a/@title').extract_first()
items['name']=i.xpath('./a/@title').get()
 
 
# -*- coding: utf-8 -*-
 
import scrapy
from ..items import TxmoviesItem
 
class TxmsSpider(scrapy.Spider):
    name = 'txms'
    allowed_domains = ['v.qq.com']
    start_urls = ['https://v.qq.com/x/bu/pagesheet/listappend=1&channel=cartoon&iarea=1&listpage=2&offse=0&pagesize=30']
    offset=0
    def parse(self, response):
        items=TxmoviesItem()
        lists=response.xpath('//div[@class="list_item"]')
        for i in lists:
            items['name']=i.xpath('./a/@title').get()
            items['description']=i.xpath('./div/div/@title').get()
            yield items
        if self.offset < 120:
            self.offset += 30
            url = 'https://v.qq.com/x/bu/pagesheet/listappend=1&channel=cartoon&iarea=1&listpage=2&offset={}&pagesize=30'.format(str(self.offset))
            yield scrapy.Request(url=url,callback=self.parse)

5.交给管道输出

# -*- coding: utf-8 -*-
# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
 
class TxmoviesPipeline(object):
    def process_item(self, item, spider):
        print(item)
        return item

6.run，执行项目

fromscrapyimportcmdline
cmdline.execute('scrapycrawltxms'.split())