爬虫爬取案例

scrapy安装

python -m pip install --upgrade pip

pip install wheel

pip install lxml

pip install twisted

pip install pywin32

pip install scrapy

1.创建项目

打开一个终端输入（默认是C盘）

scrapy startproject TXmovies

cd TXmovies

scrapy genspider txms v.qq.com

2.修改setting

ROBOTSTXT_OBEY = False
DOWNLOAD_DELAY = 1
DEFAULT_REQUEST_HEADERS = {
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language': 'en',
  'User-Agent':'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36'
}

ITEM_PIPELINES = {
   'TXmovies.pipelines.TxmoviesPipeline': 300,
}

3.确认要提取的数据，item项i

 Define here the models for your scraped items
See documentation in:
https://docs.scrapy.org/en/latest/topics/items.html
import scrapy
class TxmoviesItem(scrapy.Item):
    define the fields for your item here like:
    name = scrapy.Field()
    name = scrapy.Field()
    description = scrapy.Field()

4.写爬虫程序

import scrapy
from ..items import TxmoviesItem
class TxmsSpider(scrapy.Spider):
    name = 'txms'
    allowed_domains = ['v.qq.com']start_urls=['https://v.qq.com/x/bu/pagesheet/list?append=1&channel=cartoon&iarea=1&listpage=2&offset=0&pagesize=30']    offset=0
    def parse(self, response):
        items=TxmoviesItem()
        lists=response.xpath('//div[@class="list_item"]')
        for i in lists:
            items['name']=i.xpath('./a/@title').get()
            items['description']=i.xpath('./div/div/@title').get()
            yield items
        if self.offset < 120:
            self.offset += 30
url='https://v.qq.com/x/bu/pagesheet/list?ppend=1&channel=cartoon&iarea=1&listpage=2&offset={}&pagesize=30'.format(str(self.offset))

            yield scrapy.Request(url=url,callback=self.parse)

5.交给管道输出

Define your item pipelines here
Don't forget to add your pipeline to the ITEM_PIPELINES setting
See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
class TxmoviesPipeline(object):
    def process_item(self, item, spider):
        print(item)
        return item

6.run，执行项目

from scrapy import cmdline
cmdline.execute('scrapy crawl txms'.split())