转载请附上博文链接!
https://blog.csdn.net/qq_41562377/article/details/89791216
第一篇的猫眼电影爬取的整合性不是很强,而且整个的爬取速度较慢。
现在来一篇全猫眼电影都能爬取的,而且速度还不慢。
主要是今天在网上看到评论的一个接口,话不多说,嘿嘿嘿~~
复联四的URL
http://m.maoyan.com/mmdb/comments/movie/248172.json?
毒液的URL
http://m.maoyan.com/mmdb/comments/movie/42964.json?
大鱼
http://m.maoyan.com/mmdb/comments/movie/1202.json?
这里我要强烈推荐电影《大鱼》,真的太棒了,我敢保证,你绝对不会后悔。
提取码:mmxn
所以这次我就以《大鱼》为题,爬取它的全部评论。具体的分析之后会做,现在只是爬。
单条数据如下,和之前的所有差别,这里有cityName
{
"approve": 5,
"approved": false,
"assistAwardInfo": {
"avatar": "",
"celebrityId": 0,
"celebrityName": "",
"rank": 0,
"title": ""
},
"authInfo": "",
"avatarurl": "https://img.meituan.net/avatar/36ad06a40e9b70e4611cb630f9b0d5b673707.jpg",
"content": "《大鱼》:为了叙述一个真实感动的故事,男主人公添加了奇幻的元素作修饰,增添了童话的色彩,也一语双关地道破了儿子的对父亲的不了解!蒂姆伯顿一反过去的哥特式风格,主角也不再是德普,但是故事的衔接以及所要表达的主旨很明确:从“父亲”的身上“我”不仅学会了如何经营幸福美好的生活,而且还理解了“父亲”的苦衷以及为人处世之道的法则!在表观而言,虽是一部充满冒险的奇幻剧,而在经过了蒂姆伯顿的包装之后升华了主旨,是本片的最大亮点,也是最大的泪点,通过“父亲”每一段故事的阐述,都无时不刻不在激励每个人都应该拥有奋发图强的心态,乐观待人,冷静处事,自然会招纳更多的知心朋友!蒂姆伯顿也有煽动人心的一面,9.5分!!",
"filmView": false,
"gender": 1,
"id": 19335483,
"isMajor": false,
"juryLevel": 0,
"majorType": 0,
"movieId": 1202,
"nick": "Alex_24",
"nickName": "Alex",
"oppose": 0,
"pro": false,
"reply": 0,
"score": 4.5,
"spoiler": 0,
"startTime": "2015-06-26 13:08:42",
"supportComment": true,
"supportLike": true,
"sureViewed": 0,
"time": "2015-06-26 13:08",
"userId": 162402474,
"userLevel": 4,
"videoDuration": 0,
"vipInfo": "",
"vipType": 0
}
好了,我已经迫不及待了。
import requests
import random
import json
from datetime import datetime, timedelta
import time
class MaoYan():
def __init__(self):
#定义浏览器,让爬虫伪装成不同的浏览器,可有效降低被反爬
user_agent = ['Mozilla/5.0 (Windows NT 6.1; rv:50.0) Gecko/20100101 Firefox/50.0',
'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0',
'Mozilla/5.0 (X11; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',
'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Trident/5.0)',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14',
'Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36',
'Mozilla/5.0 (iPad; CPU OS 10_1_1 like Mac OS X) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0 Mobile/14B100 Safari/602.1',
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:50.0) Gecko/20100101 Firefox/50.0',
'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0',
'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1'
]
##定义随机函数抽取浏览器访问
num = random.randint(0, 9)
user_agent = user_agent[num]
self.headers = {'user_agent': user_agent, #伪装
'Connection': 'keep - alive', #保持连接不断
'Host': 'm.maoyan.com' #增加伪装
}
self.fp=open('maoyan.txt','a',encoding='utf-8')
def req_data(self, url):
response=requests.get(url=url, headers=self.headers,timeout=5) #5秒还没请求成功就跳过
try:
if response.status_code==200:
return response.text
else:
time.sleep(3)
self.req_data(url=url)
except ConnectionRefusedError:
pass
def get_data(self, data):
cmts=json.loads(data)['cmts']
comments = []
try:
for item in cmts:
comment = {
'userId': item.get('userId'), # 用户ID
'nick': item.get('nick'), # 用户昵称
'cityName': item.get('cityName'), #城市
'gender': item.get('gender'), # 用户性别
'content': item.get('content'), # 用户评论
'score': item.get('score'), # 用户评分
'startTime': item.get('startTime'), # 时间
'userLevel': item.get('userLevel') # 用户等级
}
comments.append(comment)
return comments
except Exception as e:
print(e)
pass
def save_data(self, info):
infos=json.dumps(info,ensure_ascii=False)
self.fp.write(infos)
self.fp.write('\n')
def main(self):
start_time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
end_time = '2003-4-25 00:00:00'
while start_time > end_time:
url='http://m.maoyan.com/mmdb/comments/movie/1202.json?_v_=yes&offset=15&startTime=' + start_time.replace(' ', '%20')
print(url)
try:
response=self.req_data(url=url)
except ConnectionRefusedError or Exception or ConnectionError:
time.sleep(2)
response=self.req_data(url=url)
else:
time.sleep(0.1)
try:
comments=self.get_data(response)
start_time = comments[14]['startTime']
print(start_time)
start_time = datetime.strptime(start_time, '%Y-%m-%d %H:%M:%S') + timedelta(seconds=-1)
start_time = datetime.strftime(start_time, '%Y-%m-%d %H:%M:%S')
for item in comments:
print(item)
self.save_data(item)
except IndexError:
time.sleep(2)
start_time = datetime.strptime(start_time, '%Y-%m-%d %H:%M:%S') + timedelta(seconds=-2)
start_time = datetime.strftime(start_time, '%Y-%m-%d %H:%M:%S')
if __name__ == '__main__':
MY = MaoYan()
MY.main()
代码是写好了,但是只爬到159条数据看了整个猫眼
差点自闭,就差一点,那么好的电影,咋地没人评论咧。
分析原因是年代久远,大鱼是2003年上映的,这时还没有猫眼吧,我猜的。
那就变一个,来个小可爱 毒液的。
变化的地方
0)self.fp=open('maoyan.txt','a',encoding='utf-8') #这里你可能要更改文件名
1)end_time = '2018-11-09 00:00:00'
2)url='http://m.maoyan.com/mmdb/comments/movie/42964.json?_v_=yes&offset=15&startTime=' + start_time.replace(' ', '%20')
3)可能要变 while start_time > end_time 里面的 大于号 转成 小于号
爬取成功,能爬9W多条,足够分析了。
谢谢大家观看。
注:只做学习使用,商业用途等自行负责。