python sdweather_爬虫框架抓取练习-JMU SD weatherspider,scrapy,jmusdWeatherSpider

python爬取中国天气网福建省当日温度

一、效果图

二、运行步骤

创建项目：scrapy startproject sdWeatherSpider

执行命令： scrapy crawl everyCityinSD ，运行爬虫程序

三、主要代码

everyCityinSD.py

# -*- coding: utf-8 -*-

import scrapy

from urllib.request import urlopen

from sdWeatherSpider.items import SdweatherspiderItem

import re

class EverycityinsdSpider(scrapy.Spider):

name = 'everyCityinSD'

allowed_domains = ['www.weather.com.cn']

start_urls = ['http://www.weather.com.cn/fujian/index.shtml']

def parse(self, response):

# 处理每个城市的天气预报页面数据

item = SdweatherspiderItem()

citys=[]

weather=[]

selector = response.xpath('//div[@class="forecastBox"]')

for dl in selector.xpath('./dl'):

city = dl.xpath('./dt//a//text()').extract()[0]

high=dl.xpath('./dd//a//span//text()').extract()[0]

low=dl.xpath('./dd//b//text()').extract()[0]

tem='high'+high+'low'+low

citys.append(city)

weather.append(tem)

# 存放天气数据

item['city'] = citys

item['weather'] = weather

return [item]

pass

有部分配置文件要修改。如果自行创建框架了，就只要替换everyCityinSD.py   ，settings.py，pipelines.py，items.py,以及画图的draw