使用requests爬取豆瓣电影

使用requests爬取豆瓣电影

一、分析

豆瓣电影信息https://movie.douban.com/explore#!type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=20&page_start=0

我们根据首页信信息获取链接

然后，点击进入二级链接

到了这个页面，我们就可以看到我们需要的信息了

接着，我们回到上个网页，查看一下我们需要爬取多少电影信息，可是当我们点击加载更多的时候，页面没有跳转，网页url也也有刷新，但是新的信息又出来了，根据经验，应该采用了异步刷新技术

通过我们对网页的分析，最终我获取的了数据加载的接口了，然后点进去，可以发现，这就是我们想要的数据。

然后我们通过对url的分析

https://movie.douban.com/j/search_subjects?type=movie&tag=热门&sort=recommend&page_limit=20&page_start=0

• page_limit=20 决定请求信息的数量
• page_start=0 决定请求的位置
  

更重要的是这里的数据都是json格式的数据，我们就可以省去页面解析的步骤了

二、写代码

1.导入对应的python库

from lxml import etree#用于对网页的数据解析
import requests#网页请求
import json#数据储存

2.构建一个网页请求方法

def requests_get(url):
    #伪装头部
    headers ={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                      "AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/54.0.2840.99 Safari/537.36"
    }
    #进行请求
    r = requests.get(url,headers=headers)
    #进行编码设置
    r.encoding = 'utf-8'
    #返回获取信息
    return r

3.创建一个解析网页信息并返回电影链接

def get_mv_listurl(html):
	#初始化一个电影链接的列表
    mvlist = []
    #由于信息是一种json的字符串
    jsondata = json.loads(html.text)['subjects']
    for data in jsondata:
        mvlist.append(data['url'])
    return mvlist

4. 创建一个对每个电影url跟进的方法，并获取有用的数据

def get_mvdata(url):
    xpathmv = etree.HTML(requests_get(url).text)
    #获取数据
    mvdata = {}
    #电影名称
    name = xpathmv.xpath('.//h1/span[1]/text()')[0]
    mvdata['name'] = name
    #评分
    ge = xpathmv.xpath('.//strong[@class="ll rating_num"]/text()')[0]
    mvdata['ge'] = ge
    #评价人数
    number = xpathmv.xpath('.//div[@class="rating_sum"]/a[@class="rating_people"]/span/text()')[0]
    mvdata['number'] =number
    #年份
    year = xpathmv.xpath('.//h1/span[2]/text()')[0][1:5]
    mvdata['year'] = year
    #导演
    director = xpathmv.xpath('.//div[@id="info"]/span[1]/span[@class="attrs"]/a/text()')
    mvdata['director'] = director
    #编剧
    writer = xpathmv.xpath('.//div[@id="info"]/span[2]/span[@class="attrs"]/a/text()')
    mvdata['writer'] = writer
    #主演
    act = xpathmv.xpath('.//div[@id="info"]/span[3]/span[@class="attrs"]/a/text()')
    #时长
    mvdata['act'] = act
    length = xpathmv.xpath('.//span[@property="v:runtime"]/text()')[0]
    mvdata['length'] = length
    return mvdata

这里的策略是通过url获取单个网页信息，然后进行xml编码，在通过xpath获取网页的有用数据

5.写一个将数据储存为文件的方法，这里储存的格式是json

def stockpile_data_json(mvdata):
    with open('douban_data.json','a',encoding='utf-8') as file:
        line = json.dumps(mvdata, ensure_ascii=False) + "\n"
        file.write(line)

6.编写main方法，并将所有方法串联起来

if __name__ == '__main__':
    #初始化一个计数单位
    n = 0
    #初始化url
    start_url = "https://movie.douban.com/j/search_subjects?type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=100&page_start="
    #判断是否获取所有的信息，大概有300多部
    while n<400:
        #构建url
        url =start_url+str(n)
        print('--------------------------',n,'------------------------------')
        n = n+100#计数增加
        html = requests_get(url)
        mv_url_list = get_mv_listurl(html)
        for mvurl in mv_url_list:
            mvdata = get_mvdata(mvurl)
            stockpile_data_json(mvdata)
            print('>>>',mvdata['name'])

三、源码

from lxml import etree
import requests
import json
def requests_get(url):
    #伪装头部
    headers ={
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                      "AppleWebKit/537.36 (KHTML, like Gecko) "
                      "Chrome/54.0.2840.99 Safari/537.36"
    }
    #进行请求
    r = requests.get(url,headers=headers)
    #进行编码设置
    r.encoding = 'utf-8'
    #返回获取信息
    return r
    
def get_mv_listurl(html):
    mvlist = []
    jsondata = json.loads(html.text)['subjects']
    for data in jsondata:
        mvlist.append(data['url'])
    return mvlist

def stockpile_data_json(mvdata):
    with open('douban_data.json','a',encoding='utf-8') as file:
        line = json.dumps(mvdata, ensure_ascii=False) + "\n"
        file.write(line)

def get_mvdata(url):
    xpathmv = etree.HTML(requests_get(url).text)
    #获取数据
    mvdata = {}
    #电影名称
    name = xpathmv.xpath('.//h1/span[1]/text()')[0]
    mvdata['name'] = name
    #评分
    ge = xpathmv.xpath('.//strong[@class="ll rating_num"]/text()')[0]
    mvdata['ge'] = ge
    #评价人数
    number = xpathmv.xpath('.//div[@class="rating_sum"]/a[@class="rating_people"]/span/text()')[0]
    mvdata['number'] =number
    #年份
    year = xpathmv.xpath('.//h1/span[2]/text()')[0][1:5]
    mvdata['year'] = year
    #导演
    director = xpathmv.xpath('.//div[@id="info"]/span[1]/span[@class="attrs"]/a/text()')
    mvdata['director'] = director
    #编剧
    writer = xpathmv.xpath('.//div[@id="info"]/span[2]/span[@class="attrs"]/a/text()')
    mvdata['writer'] = writer
    #主演
    act = xpathmv.xpath('.//div[@id="info"]/span[3]/span[@class="attrs"]/a/text()')
    #时长
    mvdata['act'] = act
    length = xpathmv.xpath('.//span[@property="v:runtime"]/text()')[0]
    mvdata['length'] = length
    return mvdata


if __name__ == '__main__':
    #初始化一个计数单位
    n = 0
    #初始化url
    start_url = "https://movie.douban.com/j/search_subjects?type=movie&tag=%E7%83%AD%E9%97%A8&sort=recommend&page_limit=100&page_start="
    #判断是否获取所有的信息，大概有300多部
    while n<400:
        #构建url
        url =start_url+str(n)
        print('--------------------------',n,'------------------------------')
        n = n+100#计数增加
        html = requests_get(url)
        mv_url_list = get_mv_listurl(html)
        for mvurl in mv_url_list:
            mvdata = get_mvdata(mvurl)
            stockpile_data_json(mvdata)
            print('>>>',mvdata['name'])