前言
爬取豆瓣的 top250 电影榜
环境
请自行 安装python 需要的包
BeautifulSoup
MySQL-python
开始
创建一个py文件 demo3.py
分析页面结构
![image](//upload-images.jianshu.io/upload_images/454212-f27ba42922ea3b7f.png?imageMogr2/auto-orient/strip|imageView2/2/w/970/format/webp)
导入一些包
![image](//upload-images.jianshu.io/upload_images/454212-cb74a6815bc2427b.png?imageMogr2/auto-orient/strip|imageView2/2/w/456/format/webp)
定义一个DouBanSpider类以及一些方法
![image](//upload-images.jianshu.io/upload_images/454212-856bf859fe611489.png?imageMogr2/auto-orient/strip|imageView2/2/w/709/format/webp)
![image](//upload-images.jianshu.io/upload_images/454212-d33dd15e69a327d3.png?imageMogr2/auto-orient/strip|imageView2/2/w/624/format/webp)
![image](//upload-images.jianshu.io/upload_images/454212-061fe3706677813d.png?imageMogr2/auto-orient/strip|imageView2/2/w/697/format/webp)
![image](//upload-images.jianshu.io/upload_images/454212-8e39b31e3c7080dd.png?imageMogr2/auto-orient/strip|imageView2/2/w/519/format/webp)
运行
```
>>> python demo3.py
```
![image](//upload-images.jianshu.io/upload_images/454212-5adf663f4accef3c.png?imageMogr2/auto-orient/strip|imageView2/2/w/530/format/webp)
创建一个用于生成chart的 py文件, showdata1.py
请自行 安装python 需要的包
pygal
MySQL-python
Flask
![image](//upload-images.jianshu.io/upload_images/454212-95be9c2aad4d0741.png?imageMogr2/auto-orient/strip|imageView2/2/w/634/format/webp)
![image](//upload-images.jianshu.io/upload_images/454212-2bbc8ed952ea2d77.png?imageMogr2/auto-orient/strip|imageView2/2/w/648/format/webp)
![image](//upload-images.jianshu.io/upload_images/454212-85adfc880aff8786.png?imageMogr2/auto-orient/strip|imageView2/2/w/519/format/webp)
![image](//upload-images.jianshu.io/upload_images/454212-b466abf65584de2a.png?imageMogr2/auto-orient/strip|imageView2/2/w/842/format/webp)
**小例子**
```
#!/usr/bin/python
#coding=utf-8
import pygal
import json
from urllib2 import urlopen # python 2 syntax
# from urllib.request import urlopen # python 3 syntax
import MySQLdb
from flask import Flask
from pygal.style import DarkSolarizedStyle
import sys,os
reload(sys)
sys.setdefaultencoding('utf8')
app = Flask(__name__)
#----------------------------------------------------------------------
@app.route('/dbmovies')
def demoDBMovies():
try:
conn = MySQLdb.connect(host='localhost',user='hejing',passwd='hejing',db='books',charset='utf8')
except Exception, e:
print e
sys.exit()
cursor = conn.cursor()
sql = "select * from books order by people DESC LIMIT 10"
cursor.execute(sql)
alldata = cursor.fetchall()
title = " This is a testing demo"
line_chart = pygal.HorizontalBar()
line_chart.title = 'Best Top 10 movies in Douban'
if alldata:
for rec in alldata:
#print rec[0], rec[1]
line_chart.add(rec[0], rec[1])
cursor.close()
conn.close()
html = """
%s%s
""" % (title, line_chart.render())
return html
#----------------------------------------------------------------------
if __name__ == '__main__':
app.run(host='127.0.0.1')
```
现在还有大礼包免费赠送
推荐一下我建的python学习交流扣扣qun:850973621,群里有免费的视频教程,开发工具、
电子书籍、项目源码分享。学习python web、python爬虫、数据分析、大数据,人工智能等
技术有不懂的可以加入一起交流学习,一起进步!
作者:2010jing
链接:https://www.jianshu.com/p/67b8aac4f93e
来源:简书
著作权归作者所有。商业转载请联系作者获得授权,非商业转载请注明出处。