基于Xpath的数据采集
目标
- 数据采集
- 数据预处理
- 数据存储
一、数据采集
1、前期准备
# 导入相关库
import pandas as pd
import requests
from lxml import etree
import time
观察规律
* https://movie.douban.com/subject/25845392/comments?start=0&limit=20&status=P&sort=new_score
* https://movie.douban.com/subject/25845392/comments?start=20&limit=20&status=P&sort=new_score
* https://movie.douban.com/subject/25845392/comments?start=40&limit=20&status=P&sort=new_score
# 批量生成网页链接
urls = []
for i in range(19):
url = 'https://movie.douban.com/subject/25845392/comments?start='+str(i*20)+'&limit=20&status=P&sort=new_score'
urls.append(url)
2、基于Xpath进行数据采集
采集《长津湖》评价,主要字段包括评分,评分标签,评论时间,评论内容,点赞数量,用户 id,用户名,入会时间,居住地点。
# 批量采集数据
movie_comments = pd.DataFrame() # 用于存储所有评论目录页的字段信息
for i in urls: # 循环上面批量生成的网页
print(i)
# 发送请求
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'
}
cookies = {'Cookie': 'bid=2Xn2TzWr_Zw; __utmz=30149280.1663724334.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __gads=ID=b7f53a5b83e5e704-22da5e63aad6007a:T=1663894139:RT=1663894139:S=ALNI_MbbFS2p96tG4R74PQrad89K2ai9VA; ll="118281"; __gpi=UID=000009d8ef520d13:T=1663894139:RT=1666575669:S=ALNI_MY4J1DYTuvimFIyxk5IinlfidSqnA; push_noty_num=0; push_doumail_num=0; dbcl2="216837743:t92rRERRJB4"; ck=zlas; __utmc=30149280; __utmv=30149280.21683; ap_v=0,6.0; _pk_ref.100001.8cb4=%5B%22%22%2C%22%22%2C1666588216%2C%22https%3A%2F%2Fmovie.douban.com%2Fsubject%2F25845392%2Fcomments%3Fstatus%3DP%22%5D; _pk_id.100001.8cb4=b67eccd466165bb3.1666582708.2.1666588216.1666583309.; _pk_ses.100001.8cb4=*; __utma=30149280.1729025570.1663724334.1666574970.1666588217.6; __utmt=1; __utmb=30149280.2.10.1666588217'}
rq = requests.get(i,headers=headers,cookies=cookies)
# 网页解析
dom = etree.HTML(rq.text)
# 提取数据
user_name = dom.xpath('//*[@id="comments"]/div/div[2]/h3/span[2]/a/text()') # 用户名
score = dom.xpath('//*[@id="comments"]/div/div[2]/h3/span[2]/span[2]/@class') # 评分
score_label = dom.xpath('//*[@id="comments"]/div/div[2]/h3/span[2]/span[2]/@title') # 评分标签
comment_time = dom.xpath('//*[@id="comments"]/div/div[2]/h3/span[@class="comment-info"]/span[@class="comment-time "]/@title') # 评论时间
comment_content = dom.xpath('//*[@id="comments"]/div/div[2]/p/span/text()') # 评论内容
number_like = dom.xpath('//*[@id="comments"]/div/div[2]/h3/span[1]/span/text()') # 点赞数量
user_href = dom.xpath('//*[@id="comments"]/div/div[2]/h3/span[2]/a/@href') # 用户详情页链接
# 创建空列表,存储循环读取的数据
user_id = []
user_time = []
address = []
# 因为用户id,入会时间,入会时间在用户详情页中
# 对二级链接(用户详情页)进行循环,读取数据
for i in user_href:
# 发送请求
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36'
}
cookies = {'Cookie': 'bid=2Xn2TzWr_Zw; __utmz=30149280.1663724334.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __gads=ID=b7f53a5b83e5e704-22da5e63aad6007a:T=1663894139:RT=1663894139:S=ALNI_MbbFS2p96tG4R74PQrad89K2ai9VA; ll="118281"; __gpi=UID=000009d8ef520d13:T=1663894139:RT=1666575669:S=ALNI_MY4J1DYTuvimFIyxk5IinlfidSqnA; push_noty_num=0; push_doumail_num=0; dbcl2="216837743:t92rRERRJB4"; ck=zlas; __utmc=30149280; __utmv=30149280.21683; ap_v=0,6.0; _pk_ref.100001.8cb4=%5B%22%22%2C%22%22%2C1666588216%2C%22https%3A%2F%2Fmovie.douban.com%2Fsubject%2F25845392%2Fcomments%3Fstatus%3DP%22%5D; _pk_id.100001.8cb4=b67eccd466165bb3.1666582708.2.1666588216.1666583309.; _pk_ses.100001.8cb4=*; __utma=30149280.1729025570.1663724334.1666574970.1666588217.6; __utmt=1; __utmb=30149280.2.10.1666588217'}
user_rq = requests.get(i,headers=headers,cookies=cookies)
# 网页解析
dom_1 = etree.HTML(user_rq.text)
# 读取数据
user_id.append(dom_1.xpath('//*[@id="profile"]/div/div[2]/div[1]/div/div/text()[1]')) # 用户id
user_time.append(dom_1.xpath('//*[@id="profile"]/div/div[2]/div[1]/div/div/text()[2]')) # 入会时间
address.append(dom_1.xpath('//*[@id="profile"]/div/div[2]/div[1]/div/a/text()')) # 入会时间
# 设置强制等待时间
time.sleep(0.5)
# 数据整理
data = pd.DataFrame({
'用户名': user_name,
'用户详情页链接': user_href,
'用户id': user_id,
'入会时间': user_time,
'居住地': address,
'评分': score,
'评分标签': score_label,
'评论时间': comment_time,
'评论内容': comment_content,
'点赞数量': number_like
})
# 数据合并
movie_comments = pd.concat([movie_comments,data])
——————————————————————————————————————————
Output:
https://movie.douban.com/subject/25845392/comments?start=0&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=20&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=40&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=60&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=80&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=100&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=120&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=140&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=160&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=180&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=200&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=220&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=240&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=260&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=280&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=300&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=320&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=340&limit=20&status=P&sort=new_score
https://movie.douban.com/subject/25845392/comments?start=360&limit=20&status=P&sort=new_score
3、初数据展示
movie_comments.shape # 查看数据大小
——————————————————————————————————————————
Output:(380, 10)
movie_comments.reset_index(drop=True, inplace=True) # 重设索引
movie_comments
4、存储数据
movie_comments.to_excel('movie_comments.xlsx',index = False)
二、数据预处理
1、读取数据
movie_comments = pd.read_excel('movie_comments.xlsx')
movie_comments.info() # 数据各列信息
——————————————————————————————————————————
Output:
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 380 entries, 0 to 379
Data columns (total 10 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 用户名 380 non-null object
1 用户详情页链接 380 non-null object
2 用户id 380 non-null object
3 入会时间 380 non-null object
4 居住地 380 non-null object
5 评分 380 non-null object
6 评分标签 380 non-null object
7 评论时间 380 non-null object
8 评论内容 380 non-null object
9 点赞数量 380 non-null int64
dtypes: int64(1), object(9)
memory usage: 29.8+ KB
2、处理表中数据类型、数据可读性问题
由于这三列数据是由列表追加得到的,在数据框展现为列表,故需转化:
# 先处理列表对象的这三列:用户id、入会时间、居住地
movie_comments['用户id'] = movie_comments['用户id'].apply(lambda x:x.replace("['\\n ","").replace(" ']",""))
movie_comments['入会时间'] = movie_comments['入会时间'].apply(lambda x:x.replace("['\\n ","").replace("加入 ']",'').replace("加入 ', '\\n ']",""))
# 同时处理了缺失值,填充为“缺失”
movie_comments['居住地'] = movie_comments['居住地'].apply(lambda x:x.replace("['","").replace("']",'').replace("[]","缺失"))
movie_comments
接下来,为方便后续的数据分析工作,需要将带有时间意义的字符串转为时间类型:
# 把带有时间意义的字符串转为时间类型
# 本身就是字符串,直接用to_datetime
movie_comments['入会时间'] = pd.to_datetime(movie_comments['入会时间'])
# 本身是一个对象,先转成字符串再转时间类型
movie_comments['评论时间'] = pd.to_datetime(movie_comments['评论时间'].apply(lambda x:str(x)))
提取有效评分:
# 因为位置固定,所以只要找出第七位即是评分
movie_comments['评分'] = movie_comments['评分'].str[7]
因后续分析工作需要分析评分与居住地的关系,但居住地中包含省份、省份+城市两种,故只提取居住地的省份以便分析:
movie_comments['居住地'] = movie_comments['居住地'].str[0:2]
以上结果展示:
3、缺失值处理
# 除了以上已填充“未知”的居住地外,无缺失值
movie_comments.isnull().sum()
——————————————————————————————
Output:
用户名 0
用户详情页链接 0
用户id 0
入会时间 0
居住地 0
评分 0
评分标签 0
评论时间 0
评论内容 0
点赞数量 0
dtype: int64
4、重复值处理
import numpy as np
# 所有值都重复的行有12个
np.sum(movie_comments.duplicated())
# 去重
movie_comments.drop_duplicates(inplace=True) # 删除所有字段都相同的行
5、异常值处理
(1)通过Excel观察发现,评分数据中存在4条空值,同时这4条数据的评论标签与评论时间填充的内容一样。决定删除这4条数据:
movie_comments['评分'].value_counts()
——————————————————————————————
Output:
5 170
3 78
4 75
2 30
1 11
- 4
Name: 评分, dtype: int64
movie_comments = movie_comments[movie_comments['评分'] != '-']
movie_comments['评分'].value_counts()
——————————————————————————————
Output:
5 170
3 78
4 75
2 30
1 11
Name: 评分, dtype: int64
(2)通过观察发现,居住地中含有字母,将其删除:
movie_comments['居住地'].value_counts()
# 将带字母的地区名字替换为空,再将其替换为“缺失”,归入缺失这一部分
movie_comments['居住地'] = movie_comments['居住地'].str.replace(r'[^\u4e00-\u9fa5]','').replace('','缺失')
movie_comments['居住地'].value_counts()
6、数据存储
movie_comments.to_excel('new_movie_comments.xlsx')