利用网络爬虫爬取中国天气网某个城市近7天的天气【Python】

继上一篇爬取豆瓣电影top250之后，这篇博客还是利用requests库和BeautifulSoup进行网络爬虫的练习，这次爬取的对象是中国天气网。

说明

1.用到的库：requests，BeautifulSoup，re。其中，requests库用于获取网页内容，BeautifulSoup用于网页解析，re正则表达式库用于对爬取内容进行匹配和搜索。
2.正常情况下BeautifulSoup就可以完成网页解析，但是中国天气网现在的风向条目变成了两种风向，是在不知道怎么用BeautifulSoup去解析，只好调用re库把两个风向给检索出来。
3.不同城市有不同城市的代码，可以根据城市的不同替换URL就好；也可以把城市代码也到数据库中，根据输入城市名字找到对应的城市代码后再进行爬取。但是这里只想知道自己城市的天气，因此URL是用西安作为例子写的。

脚本如下

'''
@Author: Guo Yingwei
@Date: 2019-07-09 10:03:55
@E-mail: willgyw@126.com
@Description: crawl weather aroune 7 days of a certain city from China Weather.
'''

import requests
from bs4 import BeautifulSoup
import re

def get_page(url):
    try:
        kv = {'user-agent':'Mozilla/5.0'}
        r = requests.get(url,headers = kv)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return '错误'

def parse_page(html, return_list):
    soup = BeautifulSoup(html, 'html.parser')
    day_list = soup.find('ul', 't clearfix').find_all('li')
    for day in day_list:
        date = day.find('h1').get_text()
        wea = day.find('p',  'wea').get_text()
        if day.find('p', 'tem').find('span'):
                hightem = day.find('p', 'tem').find('span').get_text()
        else:
                hightem = ''
        lowtem = day.find('p', 'tem').find('i').get_text() 
        #win = re.search('(?<= title=").*?(?=")', str(day.find('p','win').find('em'))).group()
        win = re.findall('(?<= title=").*?(?=")', str(day.find('p','win').find('em')))
        wind = '-'.join(win)
        level = day.find('p', 'win').find('i').get_text()
        return_list.append([date, wea, lowtem, hightem, wind, level])
    #return return_list

def print_res(return_list):
    tplt = '{0:<10}\t{1:^10}\t{2:^10}\t{3:{6}^10}\t{4:{6}^10}\t{5:{6}^5}'
    print(tplt.format('日期', '天气', '最低温', '最高温', '风向', '风力',chr(12288)))
    for i in return_list:
        print(tplt.format(i[0], i[1],i[2],i[3],i[4],i[5],chr(12288)))

def main():
    url = 'http://www.weather.com.cn/weather/101110101.shtml'
    html = get_page(url)
    wea_list = []
    parse_page(html, wea_list)
    print_res(wea_list)

if __name__ == '__main__':
    main()

结果如下

网站部分的源代码如下：

<ul class="t clearfix">
<li class="sky skyid lv3 on">
<h1>9日（今天）</h1>
<big class="png40 d02"></big>
<big class="png40 n07"></big>
<p title="阴转小雨" class="wea">阴转小雨</p>
<p class="tem">
<span>26</span>/<i>18℃</i>
</p>
<p class="win">
<em>
<span title="南风" class="S"></span>
<span title="西南风" class="SW"></span>
</em>
<i><3级</i>
</p>
<div class="slid"></div>
</li>
<li class="sky skyid lv1">
<h1>10日（明天）</h1>
<big class="png40 d01"></big>
<big class="png40 n02"></big>
<p title="多云转阴" class="wea">多云转阴</p>
<p class="tem">
<span>27</span>/<i>18℃</i>
</p>
<p class="win">
<em>
<span title="东风" class="E"></span>
<span title="南风" class="S"></span>
</em>
<i><3级</i>
</p>
<div class="slid"></div>
</li>
<li class="sky skyid lv2">
<h1>11日（后天）</h1>
<big class="png40 d02"></big>
<big class="png40 n07"></big>
<p title="阴转小雨" class="wea">阴转小雨</p>
<p class="tem">
<span>31</span>/<i>20℃</i>
</p>
<p class="win">
<em>
<span title="东风" class="E"></span>
<span title="南风" class="S"></span>
</em>
<i><3级</i>
</p>
<div class="slid"></div>
</li>
<li class="sky skyid lv3">
<h1>12日（周五）</h1>
<big class="png40 d01"></big>
<big class="png40 n00"></big>
<p title="多云转晴" class="wea">多云转晴</p>
<p class="tem">
<span>33</span>/<i>21℃</i>
</p>
<p class="win">
<em>
<span title="东风" class="E"></span>
<span title="西南风" class="SW"></span>
</em>
<i><3级</i>
</p>
<div class="slid"></div>
</li>
<li class="sky skyid lv1">
<h1>13日（周六）</h1>
<big class="png40 d02"></big>
<big class="png40 n01"></big>
<p title="阴转多云" class="wea">阴转多云</p>
<p class="tem">
<span>36</span>/<i>21℃</i>
</p>
<p class="win">
<em>
<span title="东风" class="E"></span>
<span title="东南风" class="SE"></span>
</em>
<i><3级</i>
</p>
<div class="slid"></div>
</li>
<li class="sky skyid lv2">
<h1>14日（周日）</h1>
<big class="png40 d02"></big>
<big class="png40 n01"></big>
<p title="阴转多云" class="wea">阴转多云</p>
<p class="tem">
<span>36</span>/<i>21℃</i>
</p>
<p class="win">
<em>
<span title="东风" class="E"></span>
<span title="东南风" class="SE"></span>
</em>
<i><3级</i>
</p>
<div class="slid"></div>
</li>
<li class="sky skyid lv2">
<h1>15日（周一）</h1>
<big class="png40 d01"></big>
<big class="png40 n01"></big>
<p title="多云" class="wea">多云</p>
<p class="tem">
<span>32</span>/<i>21℃</i>
</p>
<p class="win">
<em>
<span title="东风" class="E"></span>
<span title="东南风" class="SE"></span>
</em>
<i><3级</i>
</p>

博主也是刚刚学习简单的定向网络爬虫，去爬一些静态网页作为练习，由于时间和水平有限，如有纰漏，欢迎指正。