中国慕课北理工系列《爬虫笔记》
1.requests库
requests库的安装小测:
import requests
r = requests.get("http://www.baidu.com")
print(r.status_code)
若状态码返回为200,则Http请求的返回状态成功了,连接成功,若返回404或其它,则说明连接失败。r.raise_for_status()在方法内部判断r.status_code是否等于200,不需要增加额外的if语句,该语句便于利用try-except进行异常处理。
r.encoding:如果header中不存在charset,则认为编码为ISO-8859-1,r.text根据r.encoding显示网页内容
r.apparent_encoding:根据网页内容分析出的编码方式,可认为是r.encoding的备选
HTTP协议:超文本传输协议,HTTP是一个基于“请求与响应”模式的,无状态的应用层协议,HTTP协议采用URL作为定位网络资源的标识,
URL格式:http://host[:port][path]
host:合法的Internet主机域名或IP地址
port:端口号,缺省端口为80
path:请求资源的路径
例子:
爬取淘宝某产品
import requests
url = "https://item.taobao.com/item.htm?spm=a217f.8051907.312185.30.279133088N3AUL&id=597976483288"
try:
kv = {'user-agent':'Mozilla/5.0'}
r = requests.get(url,headers = kv)
r.raise_for_status()
r.encoding = r.apparent_encoding
print(r.text)
except:
print("爬取失败")
图片爬取以及保存
import requests
import os
url = "http://img0.dili360.com/ga/M01/34/3B/wKgBzFS3JmOAaL7CABRuR7HHOZE680.tub.jpg@!rw9"
root = "D://pics//"
path = root + url.split('/')[-1]
try:
if not os.path.exists(root):
os.mkdir(root)
if not os.path.exists(path):
r = requests.get(url)
with open(path,'wb') as f:
f.write(r.content)
f.close()
print("文件保存成功")
else:
print("文件已存在")
except:
print("爬取失败")
百度搜索
2.BeautifulSoup库
Beautiful Soup库的安装小测
import requests
url = "https://item.taobao.com/item.htm?spm=a217f.8051907.312185.30.279133088N3AUL&id=597976483288"
try:
kv = {'user-agent':'Mozilla/5.0'}
r = requests.get(url,headers = kv)
r.raise_for_status()
r.encoding = r.apparent_encoding
print(r.text)
except:
print("爬取失败")
demo = r.text
from bs4 import BeautifulSoup
soup = BeautifulSoup(demo,'html.parser')
print(soup.prettify())
信息标记有三种形式
内容查找方法
import requests
from bs4 import BeautifulSoup
url = "http://xueshu.baidu.com/"
r = requests.get(url)
r.raise_for_status()
r.encoding = r.apparent_encoding
demo = r.text
soup = BeautifulSoup(demo,"html.parser")
print(soup.title)#标签
print(soup.a.attrs)#属性,为字典类型
print(soup.a.attrs['target'])#a属性中对‘target’进行信息提取
print(soup.a.parent.name)#a的父本的名字
print(soup.a.string)#a的字符串信息
for child in soup.body.parents:
print(child) #遍历子孙节点
print(soup.find_all(['a','b']))
例子:最好大学排名
import requests
from bs4 import BeautifulSoup
import bs4
def getHTMLText(url):
try:
r = requests.get(url, timeout=30)
r.raise_for_status()
r.encoding = r.apparent_encoding
return r.text
except:
return ""
def fillUnivList(ulist, html):
soup = BeautifulSoup(html, "html.parser")
for tr in soup.find('tbody').children:
if isinstance(tr, bs4.element.Tag):#过滤掉tr中的非标签信息(不是tag类型的信息)
tds = tr('td')
ulist.append([tds[0].string,tds[1].string,tds[2].string])
def printUnivList(ulist, num):
tplt = "{0:^10}\t{1:{3}^10}\t{2:^10}"
print(tplt.format("排名","学校名称","总分",chr(12288)))
for i in range(num):
u=ulist[i]
print(tplt.format(u[0],u[1],u[2],chr(12288))
def main():
uinfo = []
url = 'http://www.zuihaodaxue.com/Greater_China_Ranking2019_0.html'
html = getHTMLText(url)
fillUnivList(uinfo, html)
printUnivList(uinfo, 100) # 爬虫爬取的大学数量
main()