利用xpath对豆瓣top250电影网爬虫

最新推荐文章于 2024-07-26 10:45:15 发布

ru_an12138

最新推荐文章于 2024-07-26 10:45:15 发布

阅读量629

点赞数 3

文章标签：爬虫 python

本文链接：https://blog.csdn.net/ru_an12138/article/details/140164900

版权

下面是利用xpath对豆瓣top250电影网爬虫的代码

import requests
from lxml import etree
import time

for x in range(10):

    url = 'https://movie.douban.com/top250?start={}&filter='.format(x * 25)

    headers = {
        'USer-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0'
    }

    resp = requests.get(url,headers=headers).text


    #第二步：通过xpath获取有效的信息
    html = etree.HTML(resp)

    movie_list = html.xpath('/html/body/div[3]/div[1]/div/div[1]/ol/li')#/html/body/div[3]/div[1]/div/div[1]/ol/li[1]

    for movie in movie_list:
        title = movie.xpath('./div/div[2]/div[1]/a/span[1]/text()')[0]#列表取字符串#/html/body/div[3]/div[1]/div/div[1]/ol/li[1]/div/div[2]/div[1]/a/span[1]


        rating = movie.xpath('./div/div[2]/div[2]/div/span[2]/text()')[0]

        heart = movie.xpath('./div/div[2]/div[2]/p[2]/span/text()')[0]


        print(title,rating,heart)
        time.sleep(0.5)