豆瓣评分

最新推荐文章于 2024-09-27 10:11:28 发布

zzl090303

最新推荐文章于 2024-09-27 10:11:28 发布

阅读量507

点赞数 1

分类专栏： python 文章标签： python html

本文链接：https://blog.csdn.net/zzl090303/article/details/105479686

版权

python 专栏收录该内容

2 篇文章 0 订阅

订阅专栏

豆瓣评分

分析网页结构

豆瓣评分一共有225个，每个页面有25个电影评分，
根据每一页链接（https://movie.douban.com/top250?start=0&filter=；https://movie.douban.com/top250?start=25&filter=；https://movie.douban.com/top250?start=50&filter=；）可以找到规律，for循环，如若取到225，则最后也页面无法获取，所以应该设置为226，每次取25个。代码如下：
for page in range(0,226,25):
url=‘https://movie.douban.com/top250?start=s%&filter=’
print(page)
运行结果如下：
0
25
50
75
100
125
150
175
200

导入包，请求服务器

请求服务器，导入包requests,设置浏览器代理，检查网页，requests中的get获取的，所以需要用“requests.get”，需要以文本格式输出所以要用“.text”
在这里插入图片描述
import requests
test_url=‘https://movie.douban.com/top250?start=0&filter=’
#设置浏览器代理，他是字典
headers={
‘User-Agent’:‘Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36’
}
#请求源代码
reponse=requests.get(url=test_url,headers=headers).text
print(reponse)
运行结果如下：

豆瓣电影 Top 250

<meta name="baidu-site-verification" content="cZdR4xxR7RxmM4zE" />
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="Sun, 6 Mar 2005 01:00:00 GMT">

<link rel="apple-touch-icon" href="https://img3.doubanio.com/f/movie/d59b2715fdea4968a450ee5f6c95c7d7a2030065/pics/movie/apple-touch-icon.png">
<link href="https://img3.doubanio.com/f/shire/3e5dfc68b0f376484c50cf08a58bbca3700911dc/css/douban.css" rel="stylesheet" type="text/css">
<link href="https://img3.doubanio.com/f/shire/ae3f5a3e3085968370b1fc63afcecb22d3284848/css/separation/_all.css" rel="stylesheet" type="text/css">
<link href="https://img3.doubanio.com/f/movie/8864d3756094f5272d3c93e30ee2e324665855b0/css/movie/base/init.css" rel="stylesheet">
<script type="text/javascript">var _head_start = new Date();</script>
<script type="text/javascript" src="https://img3.doubanio.com/f/movie/0495cb173e298c28593766009c7b0a953246c5b5/js/movie/lib/jquery.js"></script>
<script type="text/javascript" src="https://img3.doubanio.com/f/shire/5ecaf46d6954d5a30bc7d99be86ae34031646e00/js/douban.js"></script>
<script type="text/javascript" src="https://img3.doubanio.com/f/shire/0efdc63b77f895eaf85281fb0e44d435c6239a3f/js/separation/_all.js"></script>

<style type="text/css">

.site-nav-logo img{margin-bottom:0;}

<link rel="shortcut icon" href="https://img3.doubanio.com/favicon.ico" type="image/x-icon">

<script type="text/javascript">var _body_start = new Date();</script>
<link href="//img3.doubanio.com/dae/accounts/resources/f5f3d66/shire/bundle.css" rel="stylesheet" type="text/css">

下载豆瓣客户端

豆瓣 6.0 全新发布 ×

豆瓣

扫码直接下载

iPhone · Android

豆瓣电影

搜索：

xpath，
导入lxml包，从lxml得到etree,xpath进行节点定位。
from lxml import etree
html_etree=etree.HTML(reponse)
li=html_etree.xpath(’//*[@id=“content”]/div/div[1]/ol/li’)
print (len(li))
for item in li:
#获取电影名称
name=item.xpath(’./div/div[2]/div[1]/a/span[1]/text()’)[0]#‘.‘表示前面相同的内容
print(name)
#获取电影超链接
dy_url=item.xpath(’./div/div[2]/div[1]/a/@href’)[0]
print(dy_url)
#获取星级
xj=item.xpath(’./div/div[2]/div[2]/div/span[1]/@class’)[0]
print(xj)
#获取评分
pf=item.xpath(’./div/div[2]/div[2]/div/span[2]/text()’)[0]
print(pf)
#评价人数
m=item.xpath(’./div/div[2]/div[2]/div/span[4]/text()’)[0]
print(m)

print(name,dy_url,xj,pf,m)

print("$"*100)
运行结果如下：

肖申克的救赎
https://movie.douban.com/subject/1292052/
rating5-t
9.7
1974395人评价
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $霸王别姬 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1291546 / r a t i n g 5 - t 9.61460983 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $阿甘正传 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1292720 / r a t i n g 5 - t 9.51497788 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $这个杀手不太冷 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1295644 / r a t i n g 45 - t 9.41691633 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $美丽人生 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1292063 / r a t i n g 5 - t 9.5944874 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $泰坦尼克号 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1292722 / r a t i n g 45 - t 9.41445256 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $千与千寻 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1291561 / r a t i n g 45 - t 9.41545988 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $辛德勒的名单 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1295124 / r a t i n g 5 - t 9.5763919 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $盗梦空间 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 3541415 / r a t i n g 45 - t 9.31436154 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $忠犬八公的故事 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 3011091 / r a t i n g 45 - t 9.4993496 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $海上钢琴师 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1292001 / r a t i n g 45 - t 9.31195972 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $楚门的世界 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1292064 / r a t i n g 45 - t 9.31059839 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $三傻大闹宝莱坞 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 3793023 / r a t i n g 45 - t 9.21326342 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $机器人总动员 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 2131459 / r a t i n g 45 - t 9.3947087 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $放牛班的春天 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1291549 / r a t i n g 45 - t 9.3923406 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $星际穿越 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1889243 / r a t i n g 45 - t 9.31066796 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $大话西游之大圣娶亲 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1292213 / r a t i n g 45 - t 9.21045100 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $熔炉 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 5912992 / r a t i n g 45 - t 9.3642609 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $疯狂动物城 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 25662329 / r a t i n g 45 - t 9.21243956 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $无间道 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1307914 / r a t i n g 45 - t 9.2851093 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $龙猫 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1291560 / r a t i n g 45 - t 9.2888995 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $教父 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1291841 / r a t i n g 45 - t 9.3649741 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $当幸福来敲门 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 1849031 / r a t i n g 45 - t 9.11061614 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $怦然心动 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 3319755 / r a t i n g 45 - t 9.11230844 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ $触不可及 h t t p s : / / m o v i e . d o u b a n . c o m / s u b j e c t / 6786002 / r a t i n g 45 - t 9.2693600 人评价$ $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

汇总
import requests
from lxml import etree

#设置浏览器代理，他是字典
headers={
‘User-Agent’:‘Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36’
}
for page in range(0,226,25):
url=‘https://movie.douban.com/top250?start=s%&filter=’
#请求源代码,200表示成功
reponse=requests.get(url=url,headers=headers).text

print(reponse)

html_etree=etree.HTML(reponse)
li=html_etree.xpath('//*[@id="content"]/div/div[1]/ol/li')
# print (len(li))
for item in li:
    #获取电影名称
    name=item.xpath('./div/div[2]/div[1]/a/span[1]/text()')[0]
    #获取电影超链接
    dy_url=item.xpath('./div/div[2]/div[1]/a/@href')[0]
    #获取星级
    xj=item.xpath('./div/div[2]/div[2]/div/span[1]/@class')[0]
    #获取评分
    pf=item.xpath('./div/div[2]/div[2]/div/span[2]/text()')[0]
    #评价人数
    m=item.xpath('./div/div[2]/div[2]/div/span[4]/text()')[0]
    print(name,dy_url,xj,pf,m)
    print("*"*50)