中国慕课北理工系列《爬虫笔记》-requests库，beautifulsoup库

中国慕课北理工系列《爬虫笔记》

1.requests库

requests库的安装小测：

import requests
r = requests.get("http://www.baidu.com")
print(r.status_code)

若状态码返回为200，则Http请求的返回状态成功了，连接成功，若返回404或其它，则说明连接失败。r.raise_for_status()在方法内部判断r.status_code是否等于200，不需要增加额外的if语句，该语句便于利用try-except进行异常处理。

r.encoding:如果header中不存在charset，则认为编码为ISO-8859-1，r.text根据r.encoding显示网页内容

r.apparent_encoding:根据网页内容分析出的编码方式，可认为是r.encoding的备选

 

HTTP协议：超文本传输协议，HTTP是一个基于“请求与响应”模式的，无状态的应用层协议，HTTP协议采用URL作为定位网络资源的标识，

          URL格式：http://host[:port][path] 

          host:合法的Internet主机域名或IP地址

           port：端口号，缺省端口为80

          path：请求资源的路径   

 

例子：

爬取淘宝某产品

import requests
url =  "https://item.taobao.com/item.htm?spm=a217f.8051907.312185.30.279133088N3AUL&id=597976483288"
try:
    kv = {'user-agent':'Mozilla/5.0'}
    r = requests.get(url,headers = kv)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    print(r.text)
except:
    print("爬取失败")

图片爬取以及保存

import requests
import os
url = "http://img0.dili360.com/ga/M01/34/3B/wKgBzFS3JmOAaL7CABRuR7HHOZE680.tub.jpg@!rw9"
root = "D://pics//"
path = root + url.split('/')[-1]
try:
    if not os.path.exists(root):
        os.mkdir(root)
    if not os.path.exists(path):
        r = requests.get(url)
        with open(path,'wb') as f:
            f.write(r.content)
            f.close()
            print("文件保存成功")
    else:
        print("文件已存在")
except:
    print("爬取失败")

百度搜索

 

2.BeautifulSoup库

Beautiful Soup库的安装小测

import requests
url =  "https://item.taobao.com/item.htm?spm=a217f.8051907.312185.30.279133088N3AUL&id=597976483288"
try:
    kv = {'user-agent':'Mozilla/5.0'}
    r = requests.get(url,headers = kv)
    r.raise_for_status()
    r.encoding = r.apparent_encoding
    print(r.text)
except:
    print("爬取失败")
demo = r.text
from bs4 import BeautifulSoup
soup = BeautifulSoup(demo,'html.parser')
print(soup.prettify())

 

 

 

 

 

 

 

 

信息标记有三种形式

 

内容查找方法

 

import requests
from bs4 import BeautifulSoup
url = "http://xueshu.baidu.com/"
r = requests.get(url)
r.raise_for_status()
r.encoding = r.apparent_encoding
demo = r.text
soup = BeautifulSoup(demo,"html.parser")
print(soup.title)#标签
print(soup.a.attrs)#属性，为字典类型
print(soup.a.attrs['target'])#a属性中对‘target’进行信息提取
print(soup.a.parent.name)#a的父本的名字
print(soup.a.string)#a的字符串信息
for child in soup.body.parents:
    print(child)    #遍历子孙节点
print(soup.find_all(['a','b']))

 

 

例子：最好大学排名

import requests
from bs4 import BeautifulSoup
import bs4
def getHTMLText(url):
    try:
        r = requests.get(url, timeout=30)
        r.raise_for_status()
        r.encoding = r.apparent_encoding
        return r.text
    except:
        return ""
 
def fillUnivList(ulist, html):
    soup = BeautifulSoup(html, "html.parser")
    for tr in soup.find('tbody').children:
        if isinstance(tr, bs4.element.Tag):＃过滤掉ｔｒ中的非标签信息（不是ｔａｇ类型的信息）
            tds = tr('td')
            ulist.append([tds[0].string,tds[1].string,tds[2].string])
 
def printUnivList(ulist, num):
    tplt = "{0:^10}\t{1:{3}^10}\t{2:^10}"
    print(tplt.format("排名","学校名称","总分",chr(12288)))
    for i in range(num):
        u=ulist[i]
        print(tplt.format(u[0],u[1],u[2],chr(12288))
     
def main():
    uinfo = []
    url = 'http://www.zuihaodaxue.com/Greater_China_Ranking2019_0.html'
    html = getHTMLText(url)
    fillUnivList(uinfo, html)
    printUnivList(uinfo, 100) # 爬虫爬取的大学数量
main()