简书python爬虫权威_python爬虫入门-简书七日热门文章数据

前言

以我的理解，写一个爬虫分为以下几个步骤

分析目标网站

访问单个网页地址，获取网页源代码

提取数据

保存数据

抓取剩余网页

以下开始正题

1. 分析目标网站

目标网站为简书七日热门文章 http://www.jianshu.com/trending/weekly 。 提取数据为用户，标题，阅读量，评论量，获赞量，打赏数

提取目标

用chrome tools 查看这个网页，是用ajax加载的，分析规律，发现url为 http://www.jianshu.com/trending/weekly?page=1 ， page=1 至 page=5.

url规律

2. 访问单个网页地址，获取网页源代码

设置url

url = 'http://www.jianshu.com/trending/weekly?page=1'

设置头部信息(用来伪装请求，本案例中可省略)

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'}

request = urllib2.Request(url=url, headers=headers)

发送请求和接收响应

html = urllib2.urlopen(request)

3. 从源代码中提取数据

# 先用BeautifulSoup转换一下，以便之后解析

bsObj = BeautifulSoup(html.read(), 'lxml')

抓出每篇文章的源代码，并提取目标数据(写的很差劲，just work)

文章源码

items = bsObj.findAll("div", {"class": "content"})

for item in items:

author = item.find("a", {"class": "blue-link"}).get_text()

title = item.find("a", {"class": "title"}).get_text()

other = item.find("div", {"class": "meta"}).get_text()

pattern = re.compile('(\d+)')

content = re.findall(pattern, other)

view = content[0]

comment = content[1]

like = content[2]

money = content[3] if (len(content) == 4) else 0 # 非常不严谨，暂时这么做

4. 保存数据

with open('articlesOfSevenDays.csv', 'a') as resultFile:

wr = csv.writer(resultFile, dialect= 'excel')

wr.writerow([author,title,view,comment,like,money])

因为遇到编码问题，所以添加以下代码

import sys

reload(sys)

sys.setdefaultencoding('utf-8')

5. 抓取剩余网页

for i in range(1,6):

print "开始抓取第{}页...".format(i)

url = 'http://www.jianshu.com/trending/weekly?page={}'. format(i)

# 重复之前提取数据和保存数据的代码

完整的代码

#!/usr/bin/env python

# coding=utf-8

from urllib.request import Request,urlopen

from bs4 import BeautifulSoup

from urllib.error import HTTPError

import re

import csv

import os

def getHTML(i):

url = 'http://www.jianshu.com/trending/weekly?page={}'.format(i)

headers = {

'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36'}

try:

request = Request(url=url, headers=headers)

html = urlopen(request)

bsObj = BeautifulSoup(html.read(), 'lxml')

items = bsObj.findAll("div", {"class": "content"})

except HTTPError as e:

print(e)

exit()

return items

def getArticleInfo(items):

articleInfo= []

for item in items:

author = item.find("a", {"class": "blue-link"}).get_text()

title = item.find("a", {"class": "title"}).get_text()

other = item.find("div", {"class": "meta"}).get_text()

pattern = re.compile('(\d+)')

content = re.findall(pattern, other)

view = content[0]

comment = content[1]

like = content[2]

money = content[3] if (len(content) == 4) else 0 # 不太严谨

articleInfo.append([author, title, view, comment, like, money])

return articleInfo

dir = "../jianshu/"

if not os.path.exists(dir):

os.makedirs(dir)

csvFile = open("../jianshu/jianshuSevenDaysArticles.csv","wt",encoding='utf-8')

writer = csv.writer(csvFile)

writer.writerow(("author", "title", "view", "comment", "like", "money"))

try:

for i in range(1, 6):

items = getHTML(i)

articleInfo = getArticleInfo(items)

for item in articleInfo:

writer.writerow(item)

finally:

csvFile.close()

抓取结果

image.png

总结

页面解析水平不好，接下来要学习：正则表达式，beautifulSoup，lxml

遇到的编码问题待学习