爬取猫眼电影相关信息

最新推荐文章于 2021-12-23 14:33:00 发布

骑着蜗牛追星星呐

最新推荐文章于 2021-12-23 14:33:00 发布

阅读量367

点赞数

分类专栏： Python爬虫文章标签：爬虫

本文链接：https://blog.csdn.net/qq_43320162/article/details/89157383

版权

Python爬虫专栏收录该内容

2 篇文章 0 订阅

订阅专栏

本文介绍了一个使用Python进行猫眼电影数据爬取的实战案例，通过requests库获取网页内容，利用lxml库解析HTML，提取电影名称、导演、演员等信息。代码实现了翻页功能，展示了如何处理数据并组织成JSON格式。

摘要由CSDN通过智能技术生成

from fake_useragent import UserAgent
from lxml import etree
import requests

def get_stree(url):
‘’’
:param url: 电影页面的地址
:return: 该页面的etree
‘’’
requests.packages.urllib3.disable_warnings()
headers={‘User-Agent’:UserAgent().random}
req=requests.get(url,headers,verify=False)
e=etree.HTML(req.text)
return e

def get_movie(e):
movie_name=e.xpath(’//div[@class=“channel-detail movie-item-title”]/a/text()’)
movie_href=e.xpath(’//div[@class=“channel-detail movie-item-title”]/a/@href’)
movie = []
for i in range(len(movie_href)):
url=“https://maoyan.com”+movie_href[i]
new_e=get_stree(url)
import re
try:
movie_director=re.sub(’(\n)|( )’,"",new_e.xpath(’//ul[@class=“celebrity-list clearfix”]/li[@class=“celebrity “]/div[@class=“info”]/a/text()’)[0])
except:
movie_director=“暂无”
movie_actor=map(lambda i:i.replace(”\n”,"").replace(" “,”"),new_e.xpath(’//li[@class=“celebrity actor”]/div[@class=“info”]/a/text()’))
strs=’’
for j in movie_actor:
strs+=j+" "*3
obj={}
obj[‘name’]=movie_name[i]
obj[‘director’]=movie_director
obj[‘actor’]=strs
obj[‘url’]=url
movie.append(obj)
return movie

def get_num(num,movies=[]):
for i in range(num):
url = ‘https://maoyan.com/films?showType=3&offset=’ + str(i * 30)
movie = {}
movie[‘id’] = i+1
movie[‘type’] = ‘第{0}页数据’.format(i + 1)
movie[‘child’] = get_movie(get_stree(url))
movies.append(movie)
return movies
movies=[]
print(get_num(2,movies=movies))