python3爬取html页面内容

使用python3爬取html页面内容使用到的模块有 urllib，BeautifulSoup
使用下面代码就可以爬取网站小说的内容。但网站出于反爬考虑，把小说内容段落顺序打乱。
然后通过前端页面按一定规则拼重新排序，至于排序规则没有去研究，本文只展示爬取过程。

安装BeautifulSoup模块

pip3 install beautifulsoup4
Collecting beautifulsoup4
  Downloading beautifulsoup4-4.6.0-py3-none-any.whl (86kB)
    100% |████████████████████████████████| 92kB 199kB/s 
Installing collected packages: beautifulsoup4
Successfully installed beautifulsoup4-4.6.0

代码

#!/usr/bin/python
#coding: UTF-8
from urllib.request import Request,urlopen 
from urllib.error import URLError, HTTPError
from bs4 import BeautifulSoup

def getContent(url):
	req = Request(url)
	#增加header头信息
	req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36')
	try: 
		response = urlopen(req)
		buff = response.read()
		html = buff.decode("utf8")
		response.close()
	except HTTPError as e:
		print('The server couldn\'t ful ll the request.')
		print('Error code: ', e.code)
	except URLError as e:
		print('reason:%s' %  e.reason)
	return html

def saveContent(content, url):
	soup = BeautifulSoup(content,"html.parser")
	for link in soup.find("dl",{"id":"dir"}).find_all('a'):
		title = link.get_text()
		url = "http://www.99lib.net"+link.get("href")
		print("title：%s" % title)
		print("url：%s" % url)
		html = getContent(url)
		soup2 = BeautifulSoup(html,"html.parser")
		content = soup2.find("div",{"id":"content"}).get_text()
		print("content:%s" % content)

url = "http://www.99lib.net/book/915/"
content = getContent(url)
saveContent(content, url)