关于如何爬取豆瓣影评的内容已在博客中写过,可以参考Python—豆瓣影评的爬取(指环王)
本节中心在将每一条豆瓣影评放进EXcel中成为一条记录
其中有些许代码进行 了封装
中心步骤:
1.导入第三方库 csv ,导包方法见爬取单章小说中的——前半部分
2.在代码中导包
3将每一条数据写入其中
从爬取豆瓣影评代码中可以看出影评写入txt文件时是一条一条写进去的,相当于每一条评论内容都是列表中的一项,在获得了txt的内容之后,传入写为EXcel的函数中,然后对txt中的内容进行遍历,将每一条评论写在Excel中的每一行上。
函数如下:
全部代码如下:
import requests
from bs4 import BeautifulSoup
import random
import csv
def get_header():
#360
header1={
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
"Host":"erebor.douban.com"
}
#ie
header2={
"User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134",
"Host":"movie.douban.com"
}
#谷歌
header3={
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36",
"Host":"movie.douban.com"
}
header_list=[header1,header2,header3]
index=random.randint(0,2)
return header_list[index]
def getdata(url):
req = requests.get(url=url, headers=get_header())
print("响应码", req.status_code)
html = req.text
# print(html)
bf = BeautifulSoup(html, "lxml")
# print(bf)
div_comments = bf.find_all("div", class_="review-item")
# print(div_comments)
for item in div_comments:
# print(item)
print("+++++++++++++++++++++")
name = item.select(".name")[0].string
# print(comment_info)
# name=comment_info.string
print(name)
star = item.select(".main-title-rating")[0].get("title")
# star=comment_star.get("title")
print(star)
comment_review = item.select("div", class_="review-short")[2].text
review = comment_review.replace("\n\n", "")
review = review.replace(" ", "")
print(review)
with open("指环王豆瓣评论.txt", "a", encoding="utf-8") as file:
file.write("作者:%s\n" % name)
file.write("星级推荐:%s\n" % star)
file.write("评论内容:%s\n\n" % review)
file.close()
def setExcel(list):
with open("指环王豆瓣评论.csv","a",encoding="utf-8",newline="") as file:
csv_writer=csv.writer(file)
for item in list:
csv_writer.writerow(item)
file.close()
if __name__ == '__main__':
for i in range(0,4):
url="https://movie.douban.com/subject/1291571/reviews?start=%d"%(i*20)
contentLIst=getdata(url)
setExcel(contentLIst)
运行代码后生成csv文件: