大数据疫情监控项目（Ⅰ）—爬虫入门

最新推荐文章于 2024-05-05 02:16:40 发布

胸中有数-数分版

最新推荐文章于 2024-05-05 02:16:40 发布

阅读量1.6k

点赞数 1

本文链接：https://blog.csdn.net/Zengmeng1998/article/details/105954202

版权

文章目录

近日在小破站看到的项目，觉得有趣，便尝试动手实操，实现脑图如下，仅以此项目向逆行者表以最崇高的敬意。

使用urllib发送请求

request.urlopen()

from urllib import request
url="http://www.baidu.com"
res=request.urlopen(url)#访问url并获得响应
print(res.geturl())#获取主机地址
print(res.getcode())#获取请求状态代码
print(res.info())#获取响应头
##html.decode("utf_8")#解码
#print(html)

http://www.baidu.com
200
Bdpagetype: 1
Bdqid: 0x9f5e52db00040d25
Cache-Control: private
Content-Type: text/html;charset=utf-8
Date: Wed, 06 May 2020 07:04:29 GMT
Expires: Wed, 06 May 2020 07:04:05 GMT
P3p: CP=" OTI DSP COR IVA OUR IND COM "
P3p: CP=" OTI DSP COR IVA OUR IND COM "
Server: BWS/1.1
Set-Cookie: BAIDUID=9A5020C5FC0E7DC83ACD2300A0985750:FG=1; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BIDUPSID=9A5020C5FC0E7DC83ACD2300A0985750; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: PSTM=1588748669; expires=Thu, 31-Dec-37 23:55:55 GMT; max-age=2147483647; path=/; domain=.baidu.com
Set-Cookie: BAIDUID=9A5020C5FC0E7DC897BF8BADA39D1B3F:FG=1; max-age=31536000; expires=Thu, 06-May-21 07:04:29 GMT; domain=.baidu.com; path=/; version=1; comment=bd
Set-Cookie: BDSVRTM=0; path=/
Set-Cookie: BD_HOME=1; path=/
Set-Cookie: H_PS_PSSID=1435_31125_21083_31428_31341_31270_31463_31228_30824_31163_31474; path=/; domain=.baidu.com
Traceid: 1588748669021625601011483707200392924453
Vary: Accept-Encoding
Vary: Accept-Encoding
X-Ua-Compatible: IE=Edge,chrome=1
Connection: close
Transfer-Encoding: chunked

html=res.read()
#print(html)
html=html.decode('utf_8')
print(html)

<!DOCTYPE html><!--STATUS OK--><html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"><meta content="always" name="referrer"><meta name="theme-color" content="#2932e1"><link rel="shortcut icon" href="/favicon.ico" type="image/x-icon" /><link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" title="百度搜索" /><link rel="icon" sizes="any" mask href="//www.baidu.com/img/baidu_85beaf5496f291521eb75ba38eacbd87.svg"><link rel="dns-prefetch" href="//dss0.bdstatic.com"/>

……

使用requests发送请求

requests.get()

#引用与网络导入
import requests
url='http://www.baidu.com'

resp=requests.get(url)
print(resp.encoding)#查看编码

ISO-8859-1

print(resp.status_code)#查看状态码

html=resp.text
print(html)

<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>ç™¾åº¦ä¸€ä¸‹ï¼Œä½ å°±çŸ¥é“</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=ç™¾åº¦ä¸€ä¸‹ class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>æ–°é—»</a> <a href=http://www.hao123.com n
……

resp.encoding='utf_8'
html=resp.text
print(html)

<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>

print(resp.headers)#获取其稳定状态格式

{'Cache-Control': 'private, no-cache, no-store, proxy-revalidate, no-transform', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Content-Type': 'text/html', 'Date': 'Wed, 06 May 2020 07:04:30 GMT', 'Last-Modified': 'Mon, 23 Jan 2017 13:28:24 GMT', 'Pragma': 'no-cache', 'Server': 'bfe/1.0.8.18', 'Set-Cookie': 'BDORZ=27315; max-age=86400; domain=.baidu.com; path=/', 'Transfer-Encoding': 'chunked'}

#尝试获取大众点评的信息
url='http://www.dianping.com'
header={
    "User-Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
}
res=requests.get(url,headers=header)
print(res.encoding)
print(res.status_code)
#注意网站的反爬机制，利用header可以解决，注意

UTF-8
200

使用beautifulsoup4解析内容

将复杂HTML文档转换成一个树形结构，每个节点都是python对象

beautifulSoup(html)
获取节点find()、findall()/select()
获取属性 atrrs
获取文本 text

import requests
from bs4 import BeautifulSoup
url='http://wsjkw.sc.gov.cn/scwsjkw/gzbd/fyzt.shtml'
resp=requests.get(url)
print(resp.encoding)#查看编码

ISO-8859-1

resp.headers

{'Server': 'SCS', 'Date': 'Wed, 06 May 2020 06:59:18 GMT', 'Content-Type': 'text/html', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Expires': 'Wed, 06 May 2020 07:04:18 GMT', 'Cache-Control': 'max-age=300', 'Content-Encoding': 'gzip'}

resp.text#出现乱码要改变编码格式

'<!DOCTYPE html>\r\n<html lang="en">\r\n<head>\r\n    <meta charset="UTF-8">\r\n    <meta name="viewport" content="width=device-width, initial-scale=0.3,maximum-scale=1, user-scalable=no">\r\n    \r\n<meta name="SiteName" content="å\x9b\x9bå·\x9dç\x9c\x81å\x8d«ç\x94\x9få\x81¥åº·å§\x94å\x91\x98ä¼\x9a"/>\r\n<meta name="SiteDomain" content="wsjkw.sc.gov.cn"/>\r\n<meta ……