近期忙里偷闲,搞了几天python爬虫,基本可以实现常规网络数据的爬取,比如糗事百科、豆瓣影评、NBA数据、股票数据、天气预报等的爬取,整体过程其实比较简单,有一些HTML+CSS+DOM树等知识就很easy,我就以天气预报数据的爬取为例,整理出来。
需求:采用python爬取“天气网”指定时间段及地区的天气预报数据,并将爬取到的数据按顺序写入到本地EXCEL文件中。
环境配置:
python3.5
win7 64位
pandas(生成时间序列用到)
urllib(获取html)
BeautifulSoup(解析html)
xlsxwriter(写入到excel)
话不多说,直接干。。。
1、用到的库
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import datetime
import pandas as pd
import xlsxwriter as xlw
from urllib import request
from bs4 import BeautifulSoup as bs
2、两种生成月份区间的方法
# 法一:datetime,先转换为datetime类型,再做加减
def dateRange(start, end): # start='2014-09'
strptime, strftime = datetime.datetime.strptime, datetime.datetime.strftime
days = (strptime(end, "%Y-%m") - strptime(start, "%Y-%m")).days
datelist1 = [strftime(strptime(start, "%Y-%m") + datetime.timedelta(i), "%Y%m") for i in range(0, days, 1)]
datelist = sorted(list(set(datelist1)))
return datelist
# 法二:pandas产生时间序列
def dateRange1(start, end):
datelist1 = [datetime.datetime.strftime(x, '%Y%m') for x in list(pd.date_range(start=start, end=end))]
datelist = sorted(list(set(datelist1)))
return datelist
3、爬取天气网数据
需要注意对None值的处理,如果为None,则写入到本地excel中会出错,因为写入的时候默认为str类型,因此,代码中对None值先做判断,然后字符串处理。
# 爬取“天气网”天气预报
def getCommentsById(city, start, end): # city为字符串,year为列表,month为列表
weather_result = []
datelist = dateRange(start, end)
for i in datelist:
url = 'http://lishi.tianqi.com/' + city + '/' + i + '.html'
opener = request.Request(url)
opener.add_header('User-Agent', 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)')
req = request.urlopen(opener).read()
soup = bs(req, 'html.parser')
weather_m = soup.select('div .tqtongji2 > ul') # .表示class; ‘#tongji’表示id等价于a[id='tongji']
for i in weather_m[1:]: # 因为第一个为表头,所以筛除掉
tt = []
for j in range(6):
t = i.find_all('li')[j].string
if t is not None: # 存在None值的进行处理,否则不能写入到excel
tt.append(t)
else:
tt.append('None')
weather_result.append(tt)
return weather_result
4、将爬取到的数据写入到本地excel
xlsxwriter库负责将数据写入到excel中,效率非常高,支持流式写入,缓解内存效率不足。
# list数据写入到本地excel中
def list_to_excel(weather_result, filename):
workbook = xlw.Workbook('E:\%s.xlsx' % filename)
sheet = workbook.add_worksheet('weather_report')
title = ['日期', '最高气温', '最低气温', '天气', '风向', '风力']
for i in range(len(title)):
sheet.write_string(0, i, title[i], workbook.add_format({'bold': True})) # 写入表头,字体加粗
row, col = 1, 0
for a, b, c, d, e, f in weather_result:
sheet.write_string(row, col, a)
sheet.write_string(row, col + 1, b)
sheet.write_string(row, col + 2, c)
sheet.write_string(row, col + 3, d)
sheet.write_string(row, col + 4, e)
sheet.write_string(row, col + 5, f)
row += 1
workbook.close()
5、大功告成,直接去E盘查看
if __name__ == '__main__':
data = getCommentsById('jinan', '2016-05', '2017-07')
list_to_excel(data, '济南天气201605-201707')
脚注
若有错误,请指正交流。
蹄疾步稳,贴地而行,以行践言!—zlg358 2017/8/23凌晨