爬虫框架scrapy练习-jmu-sdWeatherSpider

python爬取中国天气网福建省当日温度

一、效果图

二、运行步骤

创建项目：scrapy startproject sdWeatherSpider

执行命令： scrapy crawl everyCityinSD ，运行爬虫程序

三、主要代码

everyCityinSD.py

# -*- coding: utf-8 -*-
import scrapy
from urllib.request import urlopen
from sdWeatherSpider.items import SdweatherspiderItem
import re


class EverycityinsdSpider(scrapy.Spider):
    name = 'everyCityinSD'
    allowed_domains = ['www.weather.com.cn']
    start_urls = ['http://www.weather.com.cn/fujian/index.shtml']

    def parse(self, response):
        # 处理每个城市的天气预报页面数据
        item = SdweatherspiderItem()

        citys=[]
        weather=[]
        selector = response.xpath('//div