*本文仅就相关技术细节进行学术交流,请勿用作他途*
如何获取网页中发布的位置信息,在地图上进行标记可视化?
- Step 1: 数据获取
- Step2: 地址解析
- Step3: 地图可视化
地址编码后的数据导入tableau得到的地图如下(可放大):
Step 1: 数据爬取
我们采用requests进行网页爬取,利用BeautifulSoup和re进行整理和信息抽取。代码如下:
#libraries
from bs4 import BeautifulSoup
import requests
import re
import pandas as pd
import time
# define a function of crawing html
def geturl(url):
r = requests.get(url)
r.encoding = r.apparent_encoding
return r.text
# data crawing from the public notice
url = 'https://mp.weixin.qq.com/s/MkKsQkgvUWbwj8z9jG_Zng'
text = geturl(url) #get info from original link
file = open("shanghai0404.txt",'w') #save as a new file
file.write(text) #write info
# parsing
soup = BeautifulSoup(text, "html.parser")
info = soup.find('div', 'rich_media_content')
x = str(info)
# get the infection data as a list
addresses = re.findall('</span></p><p>.*?<span style="font-size: 16px;">([^已<2022].+?)?[,。、<]+?', x)
file = open("shanghai0404_addresses.txt",'w') #save as a new file
for i in addresses:
file.write(i+'\r') #write info
# get the summary data of all districts
districts = re.findall('2022年4月4日,(.+?区)[无]?新增([\d]+?)?.*?确诊病例,新增([\d]+?)例.*?无症状感染者',x)
li = []
for item in districts:
di = []
di.extend(item)
li.append(di)
df = pd.DataFrame(li,columns=['districts','positive','asymptomatic'])
df.to_csv('shanghai2022_04_04_districts.csv')
print(addresses)
print(districts)
print(df)