萌新的Python学习日记 - 爬虫无影 - 爬取豆瓣电影top250并入库：豆瓣电影top250

博客第九天

测试页面：

豆瓣电影 Top 250

https://movie.douban.com/top250?start=0&filter=

目的：抓取该页面中每部电影的名称，链接，评分，评语

工程内容：Python3.5，jupyter notebook

工具包：requests,BeautifulSoup

代码（可翻页）：

import requests
from bs4 import BeautifulSoup as bs
import pymongo


client = pymongo.MongoClient('localhost',27017)
doubantop = client['doubantop']
toplist = doubantop['list']




# url = 'https://movie.douban.com/top250?start={}&filter='.format(i)
# url = 'https://movie.douban.com/top250?start=0&filter='
def getlist(url):
    web = requests.get(url)
    soup = bs(web.text,'lxml')
    names = soup.select('#content > div > div.article > ol > li > div > div.info > div.hd > a > span')
    links = soup.select('#content > div > div.article > ol > li > div > div.info > div.hd > a')
    votes = soup.select('#content > div > div.article > ol > li > div > div.info > div.bd > div > span.rating_num')
    descs = soup.select('#content > div > div.article > ol > li > div > div.info > div.bd > p.quote > span')
    for name,link,vote,desc in zip(names,links,votes,descs):
        data = {
            'name':name.text,
            'link':link.get('href'),
            'vote':vote.text,
            'desc':desc.text
        }
        toplist.insert_one(data)
        print(data)
# getlist(url)
for i in range(0,10):
    url = 'https://movie.douban.com/top250?start={}&filter='.format(i * 25)

    getlist(url)

print内容（部分）：

数据库内容：