从零开始学scrapy(python3版本)一

环境:
window10;python 3.6.2;scrapy 1.4.0
系统已安装Python2,python3 共存模式
python2,3版本共存以及使用问题的记录

  • 创建项目
    由于scrapy官网的示例站 dmoz.org 403了,所以先拿美剧天堂的网站练手
    我的项目工程路径在D:\workspaces\python\scrapy
    打开cmd命令行工具
cd /d D:\workspaces\python\scrapy
python3 -m scrapy startproject tutorial
cd tutorial
python3 -m scrapy genspider meijutt meijutt.com
  • 编写爬虫脚本,此时工程路径下已经自动创建了

D:\workspaces\python\scrapy\tutorial\tutorial\spiders\meijutt.py

import scrapy
from tutorial.items import MeijuttItem

class MeijuttSpider(scrapy.Spider):
    name = 'meijutt'
    allowed_domains = ['meijutt.com']
    start_urls = ['http://www.meijutt.com/new100.html']

    def parse(self, response):
        items = []
        for sel in response.xpath('//ul[@class="top-list  fn-clear"]/li'):
            item = MeijuttItem()
            item['storyName'] = sel.xpath('./h5/a/text()').extract()
            item['storyState'] = sel.xpath('./span[1]/font/text()').extract()
            if item['storyState']:
                pass
            else:
                item['storyState'] = sel.xpath('./span[1]/text()').extract()
            item['tvStation'] = sel.xpath('./span[2]/text()').extract()
            if item['tvStation']:
                pass
            else:
                item['tvStation'] = [u'未知']
            item['updateTime'] = sel.xpath('./div[2]/text()').extract()
            if item['updateTime']:
                pass
            else:
                item['updateTime'] = sel.xpath('./div[2]/font/text()').extract()
            items.append(item)
        return items
  • 设置爬取数组
D:\workspaces\python\scrapy\tutorial\tutorial\items.py
import scrapy


class TutorialItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    pass
class MeijuttItem(scrapy.Item):
    # define the fields for your item here like:
    storyName = scrapy.Field()
    storyState = scrapy.Field()
    tvStation = scrapy.Field()
    updateTime = scrapy.Field()
  • 对爬取数据进行处理
D:\workspaces\python\scrapy\tutorial\tutorial\pipelines.py
import time
import sys
import importlib
importlib.reload(sys)

class TutorialPipeline(object):
    def process_item(self, item, spider):
        return item
class MeijuttPipeline(object):
    def process_item(self, item, spider):
        today = time.strftime('%Y%m%d',time.localtime())
        fileName = today + 'movie.txt'
        with open(fileName,'a') as fp:
            fp.write(item['storyName'][0] + '\t' + str(item['storyState'][0]) + '\t' + str(item['tvStation'][0]) + '\t' + str(item['updateTime'][0]) + '\n')
        return item
  • 运行爬虫
D:\workspaces\python\scrapy\tutorial>python3 -m scrapy crawl meijutt
  • 0
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值