【python】爬虫爬取豆瓣电影信息

首先是获取网页：可以直接到豆瓣这个网页，查看网页的源代码。因为是静态的文本，所以就是可以直接在网页代码上面找到相关内容。然后就是找到爬虫需要的请求头。

看到头部分，然后将其curl（bash）部分复制下来，到https://curl.trillworks.com/
生成请求头。

然后解析网页：soup = BeautifulSoup(response.text, ‘html.parser’)
然后提取信息：
首先找到想要的信息的位置

复制他的selector例如
《肖申克的救赎》
#content > div > div.article > ol > li:nth-child(1) > div > div.info > div.hd > a > span:nth-child(1)
找到评价，导演得到想要的信息，依次复制selector
再然后进行数据的处理：
将提取的元素进行分解
a[i] = a[i].text
b[i] = b[i].text
c[i] = c[i].text
最后就是文件的写入：
fo.write(a[i])
fo.write(b[i])
fo.write(c[i])

爬虫代码：

import requests
from bs4 import BeautifulSoup
cookies = {
    'bid': 'E4gkXiSwBrM',
    'douban-fav-remind': '1',
    'll': '118088',
    '__yadk_uid': 'hkoioWDL1agCeIP3VyFfyXTl6SBJ1uf3',
}
headers = {
    'Connection': 'keep-alive',
    'Cache-Control': 'max-age=0',
    'DNT': '1',
    'Upgrade-Insecure-Requests': '1',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36',
    'Sec-Fetch-User': '?1',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'navigate',
    'Referer': 'https://movie.douban.com/top250?start=225&filter=',
    'Accept-Encoding': '',
    'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7',
}
params = (
    ('start', '0'),
    ('filter', ''),
)
fo = open("豆瓣Movies.txt",'a',encoding="utf-8")
for k in range(1):
    i=1
    response = requests.get('https://movie.douban.com/top250?start='+str(i)+'&filter=', headers=headers)
    response.encoding='utf-8'
    soup = BeautifulSoup(response.text, 'html.parser')
    title = "#content > div > div.article > ol > li > div > div.info > div.hd > a"
    dirtor = "#content > div > div.article > ol > li > div > div.info > div.bd > p:nth-child(1)"
    quote='#content > div > div.article > ol > li > div > div.info > div.bd > p.quote'
    a=soup.select(title)
    b=soup.select(dirtor)
    c=soup.select(quote)
    for i in range(0,len(a)):
        a[i] = a[i].text
        b[i] = b[i].text
        c[i] = c[i].text
        fo.write(a[i])
        fo.write(b[i])
        fo.write(c[i])
        print(" 电影名字 ：{}\n 电影导演：{}\n 电影引言: {}\n ".format(a[i],b[i],c[i]),end="")
fo.close()

借鉴于：生布冯时