python 爬取豆瓣top100电影页面

最新推荐文章于 2024-06-24 19:59:02 发布

ximu VS code

最新推荐文章于 2024-06-24 19:59:02 发布

阅读量765

点赞数

分类专栏： python 文章标签： python

本文链接：https://blog.csdn.net/An_0330/article/details/116274226

版权

python 专栏收录该内容

17 篇文章 2 订阅

订阅专栏

python 爬取豆瓣top100电影页面

运行结果截图：
在这里插入图片描述

代码：
（原网站：https://movie.douban.com/top250）

1、将页面保存避免多次访问
（登陆状态需页面cookie）

import requests

url="https://movie.douban.com/top250"
headers={'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46','cookie':'bid=sO6ny0NVNqQ; _pk_id.100001.4cf6=77175839ae9b2c3b.1619686194.1.1619686194.1619686194.; _pk_ses.100001.4cf6=*; ap_v=0,6.0; __utma=30149280.35012747.1619686195.1619686195.1619686195.1; __utmb=30149280.0.10.1619686195; __utmc=30149280; __utmz=30149280.1619686195.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utma=223695111.351459569.1619686195.1619686195.1619686195.1; __utmb=223695111.0.10.1619686195; __utmc=223695111; __utmz=223695111.1619686195.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __yadk_uid=URY22xM5zXDRrprNmeoAzbBY6gxEoqR9; __gads=ID=06784ca21b6f77c2-22a7788cafc70011:T=1619686197:RT=1619686197:S=ALNI_MbW9cpCEeKsCd89PFHC4T-1d80nKg'}
r = requests.get(url,headers=headers)
r.encoding = r.apparent_encoding
html= r.text

#print(html)

with open("maoyan250_page1.html","w+",encoding="utf-8") as f:
    f.write(html)


print("爬取完毕！")

2、爬取第一页

import requests
from bs4 import BeautifulSoup

f=open("maoyan250_page1.html",encoding="utf-8")
html=f.read()
#print(html)

soup=BeautifulSoup(html,"lxml")

movies = soup.select('ol li')
for movie in movies:
    rank = movie.find(name = 'em').get_text()
    filmname = movie.find(name = "span",class_='title').get_text()
    star = movie.find_all("p")[0].text.strip()  #用strip去除两边空格。
    star_split = star.split("主演:")     #使用回车(\n)，切割information为两段
    daoyan = star_split[0]     # 第一段为导演和演员的信息
    zhuyan = star_split[1].split("\n")[0]     # 第二段为年份、地点、类型的信息
    score = movie.find(class_="rating_num").get_text()
    pic = movie.find(name='img').get('src')
    path = filmname+".jpg"
    pic_r = requests.get(pic)
    with open(path,"wb")as f:
        f.write(pic_r.content)
    print(rank,filmname,daoyan,zhuyan,score)
        
    content=[rank,filmname,daoyan,zhuyan,score]   #将爬取内容存储为txt
    with open("douban.txt","a+",encoding="utf-8") as file:
        file.write(str(content))
        
print("第一页爬取完毕！文件写入完毕！")