python爬取微信小程序(实战篇)_python爬取猫眼的前100榜单并展示在微信小程序

首先分析要爬取的网页，对其结构及数据获取方式解析后，可采用正则筛选自己要的数据

猫眼榜单.png

import requests

from requests.exceptions import RequestException

def get_one_page(url):

try:

response = requests.get(url)

if response.status_code == 200:

return response.text

return None

except RequestException:

return None

def main():

url='http://maoyan.com/board/4'

html=get_one_page(url)

print(html)

if __name__ == '__main__':

main()

结果是猫眼那边禁止了爬虫，

解决办法 模拟浏览器请求获取

禁止访问

def get_one_page(url):

try:

headers ={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}

response = requests.get(url,headers=headers)

if response.status_code == 200:

return response.text

return None

except RequestException:

return None

成功获取网页结构数据

知乎通过正则匹配自己要的内容, 且保存为json格式文件

import requests

from requests.exceptions import RequestException

import re

import json

def get_one_page(url):

try:

headers ={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}

response = requests.get(url,headers=headers)

if response.status_code == 200:

return response.text

return None

except RequestException:

return None

def parse_one_page(html):

pattern = re.compile('

.*?board-index.*?>(\d+).*?data-src="(.*?)".*?name">

+'.*?>(.*?)

.*?star">(.*?).*?releasetime">(.*?)'

+'.*?integer">(.*?).*?fraction">(.*?).*?

',re.S)

items=re.findall(pattern,html)

for item in items:

yield{

'index':item[0],

'image':item[1],

'title':item[2],

'actor':item[3].strip()[3:],

'time':item[4].strip()[5:],

'score':item[5]+item[6]

}

def write_to_file(content):

with open('result.json','a',encoding='utf-8')as f:

f.write(json.dumps(content,ensure_ascii=False)+','+'\n')

f.close()

def main(offset):

url='http://maoyan.com/board/4?offset='+str(offset)

html=get_one_page(url)

for item in parse_one_page(html):

print(item)

write_to_file(item)

if __name__ == '__main__':

for i in range(10):

main(i*10)

获取的部分数据如下：

{

"index": "1",

"image": "http://p1.meituan.net/movie/20803f59291c47e1e116c11963ce019e68711.jpg@160w_220h_1e_1c",

"title": "霸王别姬",

"actor": "张国荣,张丰毅,巩俐",

"time": "1993-01-01(中国香港)",

"score": "9.6"

},

{

"index": "2",

"image": "http://p0.meituan.net/movie/__40191813__4767047.jpg@160w_220h_1e_1c",

"title": "肖申克的救赎",

"actor": "蒂姆·罗宾斯,摩根·弗里曼,鲍勃·冈顿",

"time": "1994-10-14(美国)",

"score": "9.5"

},

{

"index": "3",

"image": "http://p0.meituan.net/movie/23/6009725.jpg@160w_220h_1e_1c",

"title": "罗马假日",

"actor": "格利高利·派克,奥黛丽·赫本,埃迪·艾伯特",

"time": "1953-09-02(美国)",

"score": "9.1"

}

接下来，配置好小程序结构如下

小程序结构图

movie.js

var moviesData = require('../../data/result.js')

Page({

data: {

},

onLoad: function (options) {

this.setData({

movieList: moviesData.movielist

});

},

})

movie.wxml

电影排行榜前100名

{{item.title}}

主演：{{item.actor}}

上映时间：{{item.time}}

{{item.score}}

movie.wxss

.total-title{

width: 9rem;

margin: 0.5rem auto;

color: orangered;

}

.movie-box{

display: flex;

margin-top: 1.5rem;

}

.movie-index{

width: 2rem;

margin-right: 1rem;

}

.movie-image{

display: block;

width:5rem;

height:5rem;

margin-left: 0.5rem;

}

.movie-middle-box{

margin-left: 1rem;

width: 11rem;

padding-top: 0.4rem;

border-bottom: 1px solid gray

}

.movie-title{

font-size: 0.9rem;

}

.movie-actor{

font-size: 0.7rem;

}

.movie-time{

font-size: 0.7rem;

color: gray;

}

.movie-score{

color: orange;

padding-top: 1.2rem;

position: absolute;

right: 0.6rem;

}

效果如下图

小程序效果

完工 ！！