一、介绍
本文介绍爬虫的一个小用法,爬取天气预报
写爬虫时最好更改一下User-Agent,不然会被阻止访问。
本文用正则匹配获取天气预报的数据,并使用matplotlib进行绘图,然后进行分析天气预报的数据
二、具体代码
import requests
from bs4 import BeautifulSoup
import re
import pandas as pd
import matplotlib.pyplot as plt
url = 'https://www.tianqi.com/guangdong-guilin/28722/30/'
headers = {
"User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/114.0'
}
response = requests.get(url, headers=headers)
response.encoding = 'utf-8'
html = response.text
soup = BeautifulSoup(html, 'html.parser') #解析文档,html.parser为HTML解析器
weather = []
tem = []
for tag in soup.find_all('li',attrs={"class":"temp"}):
if not bool(re.search(r'\d', tag.get_text())):
tag_text = tag.get_text()
weather.append(tag_text)
else:
tag_text = tag.get_text()
tem.append(tag_text)
tem_str = ''.join(tem)
max_tem = re.findall(r'\d\d~', tem_str)
min_tem = re.findall(r'~\d\d', tem_str)
max_sum = 0
min_sum = 0
for i in range(len(max_tem)):
max_tem[i] = int(max_tem[i].replace('~', ''))
max_sum += max_tem[i]
for i in range(len(min_tem)):
min_tem[i] = int(min_tem[i].replace('~', ''))
min_sum += min_tem[i]
# 存储至excel表格中
# info = {'天气': weather, '温度': tem}
# dm_file = pd.DataFrame(info)
# dm_file.to_excel('weather.xlsx', sheet_name="数据分析")
# 绘制折线图
avg_max_tem = max_sum/len(max_tem)
avg_min_tem = min_sum/len(min_tem)
fig, ax = plt.subplots()
plt.title("Temperature changes in Guilin in the next 30 days")
plt.xlabel("Future days/day")
plt.ylabel("Celsius/℃")
day = []
for i in range(30):
day.append(i)
ax.plot(day, max_tem)
ax.plot(day, min_tem)
ax.scatter(day, max_tem)
ax.scatter(day, min_tem)
ax.axhline(y=avg_max_tem, linestyle='--', color='black')
ax.axhline(y=avg_min_tem, linestyle='--', color='black')
plt.show()
三、总结
爬虫的能力很强大,本文只是简单的运用。一定要注意不能非法运用爬虫技术,要遵纪守法。