02Python外部数据获取_下载网页

最新推荐文章于 2024-07-25 08:08:29 发布

王小小小草

最新推荐文章于 2024-07-25 08:08:29 发布

阅读量4.5k

点赞数

分类专栏： python数据分析文章标签： python

本文链接：https://blog.csdn.net/sinat_33761963/article/details/53884767

版权

本文档介绍了Python爬虫的URL下载模块，特别是使用urllib2库来下载网页。通过示例展示了如何通过URL直接下载、Request访问以及结合cookie进行网页访问，以百度网为例，解释了访问成功时HTTP状态码200的含义。

摘要由CSDN通过智能技术生成

爬虫简介

爬虫的基本结构分成3个模块：

1.URL管理模块

对计划爬取的或者已经爬取的URL进行管理。防止重复爬取或者循环爬取。

实现方式：python的set数据格式；数据库中的数据表；缓存数据库redis。

2.URL下载模块

将URL对应的网页下载到本地或者读入内存中

实现方式：

python官方的基础模块：urllib2可以用来下载网页
request或者其他第三方库

3.网页解析模块

从已经下载下来的网页中爬取数据

实现方式：

正则表达式去匹配想提取的内容
html.parser
BeautifulSoup
lxml

本文档主要来详细讲解第2个模块：URL下载模块

使用urllib2库下载网页

我们用百度网来做例子，用三种方式去下载https://www.baidu.com/的网页内容

# Python 3.x
# from urllib.request import urlopen 

#Python 2.x
import urllib2

test_url = "http://www.baidu.com/"

1.通过URL下载

urllib2.irlopen()传入参数为指定的url。

.getcode()打印出来如果是200，表示访问成功

.read()是将网页下载下来

# 通过URL下载
response = urllib2.urlopen(test_url)
print response.getcode() # 200 表示访问成功
print response.read()

200
<!DOCTYPE html><html><head><meta http-equiv="content-type" content="text/html;charset=utf-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="referrer" content="never"><title>百度一下，你就知道</title><style>*{margin:0;padding:0}html,body{font:12px arial;background:#fff;height:100%;overflow-y:auto}body,form{position:relative}img{border:0}a{color:#00c}a:active{color:#f60}input{border:0;padding:0}#wrapper{min-width:810px;height:100%;min-height:600px;position:relative}#head{position:relative;height:100%;min-height:600px;text-align:center}#head .head_wrapper{height:100%}.s_form{position:relative;top:38.2%}.s_form_wrapper{position:relative;top:-191px}#form{margin:22px auto 0;width:641px;text-align:left;z-index:100}.s_ipt_wr{border:1px solid #b6b6b6;background:#fff;display:inline-block;vertical-align:top;width:539px;height:34px;border-right-width:0;border-color:#b8b8b8 transparent #ccc #b8b8b8;overflow:hidden}.s_ipt_wr:hover{border-color:#999 transparent #b3b3b3 #999}.s_ipt_wr.iptfocus{border-color:#4791ff transparent #4791ff #4791ff}.s_ipt{width:526px;height:22px;font:16px/18px arial;line-height:22px\9;margin:6px 0 0 7px;background:transparent;border:0;outline:0}.s_btn_wr{display:inline-block;*position:relative;z-index:0;vertical-align:top;width:auto;height:auto;border-bottom:1px solid transparent;*border-bottom:0}.s_btn{padding-top:2px\9;cursor:pointer;width:100px;height:36px;color:white;font-size:15px;letter-spacing:1px;background:#3385ff;border-bottom:1px solid #2d78f4;outline:medium;*border-bottom:0}.s_btn:hover{background:#317ef3;border-bottom:1px solid #2868c8;*border-bottom:0;box-shadow:1px 1px 1px #ccc}#kw{position:relative}.bdsug{width:538px;position:absolute;top:36px;display:none;border:1px solid #ccc;border-top:0;box-shadow:1px 1px 3px #ededed}.bdsug.bdsugbg{background:url(https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/home/img/sugbg_6a9201c2.png) 100% 100% no-repeat #fff;background-size:100px 110px;background-image:url(https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/home/img/sugbg_d24a0811.gif)\9}.bdsug li{width:522px;color:#000;list-style:none;font-size:14px;line-height:25px;padding:0 8px;position:relative;cursor:default}.bdsug li.bdsug-s{background:#f0f0f0}#u1{position:absolute;right:0;top:0;margin:19px 0 5px 0;padding:0 96px 0 0}#u1 a,#u1 a:visited{float:left;color:#333;font-weight:bold;line-height:24px;margin-left:20px;font-size:13px;text-decoration:underline}#u1 a:hover,#u1 a:active{text-decoration:underline;color:#00c}#u1 a.pf,#u1 a.pf:visited,#u1 a.lb,#u1 a.lb:visited{font-weight:normal}#u1 a.bri,#u1 a.bri:visited{display:inline-block;position:absolute;font-weight:normal;text-decoration:none;right:10px;width:60px;height:24px;color:white;background:#38f;line-height:24px;font-size:13px;text-align:center;margin-right:2px}#u1 a.bri:hover,#u1 a.bri:active{color:#fff;text-decoration:underline}#ftCon{height:100px;position:absolute;bottom:44px;text-align:center;width:100%}#ftConw{width:720px;margin:0 auto}#lh{margin:16px 0 5px;word-spacing:3px}#lh a{margin:0 10px}#cp,#cp a{color:#666}#cp .c-icon-icrlogo{width:14px;height:17px;display:inline-block;background:url(https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/img/icons_3bfb8e45.png) -600px -96px no-repeat;position:relative;top:3px}</style><!--[if lte IE 8]><style>#head{height:480px\9;z-index:100}.s_form{top:260px}</style><![endif]--><noscript><meta http-equiv="refresh" content="0;url=/baidu.html?from=noscript"/></noscript></head><body><div id="wrapper"><div id="head"><div class="head_wrapper"><div class="s_form"><div class="s_form_wrapper"><div id="lg"><img src="https://www.baidu.com/img/bd_logo1.png" width="270" height="129"></div><form id="form" name="f" action="http://www.baidu.com/s" class="fm"><div class="bdsug bdsugbg" id="bdsug"></div><span class="bg s_ipt_wr" id="s_ipt_wr"><input id="kw" name="wd" class="s_ipt" value="" maxlength="100" autocomplete="off"></span><span class="bg s_btn_wr" id="s_btn_wr"><input type="submit" id="su" value="百度一下" class="bg s_btn"></span></form></div></div><div id="u1"><a href="http://news.baidu.com" name="tj_trnews" class="mnav">新闻</a><a href="http://www.hao123.com" name="tj_trhao123" class="mnav">hao123</a><a href="http://map.baidu.com" name="tj_trmap" class="mnav">地图</a><a href="http://v.baidu.com" name="tj_trvideo" class="mnav">视频</a><a href="http://tieba.baidu.com" name="tj_trtieba" class="mnav">贴吧</a><a href="https://passport.baidu.com/v2/?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2F" class="lb">登录</a><a href="http://www.baidu.com/gaoji/preferences.html" name="tj_settingicon" class="pf">设置</a><a href="http://www.baidu.com/more/" name="tj_briicon" class="bri" >更多产品</a></div></div></div> <div id="ftCon"> <div id="ftConw"> <p id="lh"> <a id="setf" href="http://www.baidu.com/cache/sethelp/index.html" target="_blank">把百度设为主页</a><a href="http://home.baidu.com">关于百度</a><a href="http://ir.baidu.com">About Baidu</a></p><p id="cp">&copy;2015&nbsp;Baidu&nbsp;<a href="http://www.baidu.com/duty/" name="tj_duty">使用百度前必读</a>&nbsp;京ICP证030173号&nbsp;<i class="c-icon-icrlogo"></i></p></div></div></div></body></html><script>eval(function(p,a,c,k,e,r){e=function(c){return(c<62?'':e(parseInt(c/62)))+((c=c%62)>35?String.fromCharCode(c+29):c.toString(36))};if("0".replace(0,e)==0){while(c--)r[e(c)]=k[c];k=[function(e){return r[e]||e}];e=function(){return"[2-9t-zA-R]"};c=1};while(c--)if(k[c])p=p.replace(new RegExp("\\b"+e(c)+"\\b","g"),k[c]);return p}("!2(o){5 p=document,k=p.body,g=-1,l=\"\",m=2(a){return p.getElementById(a)},i=2(b,c,d){try{b.attachEvent(\"on\"+c,2(){d.H(b,o.event)})}catch(e){b.addEventListener(c,2(a){d.H(b,a)},false)}},q=2(a){5 b=p.createElement(\"script\");b.charset=\"gbk\";4(b.w){b.I=2(){4(b.w==\"loaded\"||b.w==\"complete\"){b.I=null;k.J(b)}}}8{b.onload=2(){k.J(b)}}b.src=a;k.insertBefore(b,k.firstChild)},j=m(\"kw\"),n=m(\"s_ipt_wr\"),s=m(\"form\"),h=m(\"6\"),f=h.getElementsByTagName(\"7\"),r=\"https://sp0.baidu.com/5a1Fazu8AA54nxGko9WTAnF6hhy/su?json=1&cb=K&wd=\";o.K=2(a){4(a.s&&a.s.9){x(5 b='<L>',c=0,d=a.s.9>=10?10:a.s.9;c<d;c++){b+='<7 y-z=\"'+a.s[c]+'\" class=\"6-overflow\">'+a.s[c]+'</7>'}b+='</L>';h.innerHTML=b;h.A.B=\"block\"}8{h.A.B=\"M\"}};i(o,'load',2(){j.N()});i(j,\"N\",2(){n.3=n.3+\" iptfocus\";q(r+O(u.t))});i(j,\"blur\",2(){n.3=n.3.v(/\\siptfocus/,\"\");h.A.B=\"M\"});i(j,\"keyup\",2(a){5 b=f.9,c=u.t;4(a.C==\"P\"||a.C==\"Q\"){!l&&(l=c);4(a.C==\"P\"){--g<-1?g=b-1:\"\"}8 4(a.KeyCode=\"Q\"){++g>b-1?g=-1:\"\"}x(5 d=0;d<b;d++){f[d].3=f[d].3.v(/6\\-s/,\"\")}4(g!=-1){f[g].3=f[g].3+\" 6-s\";u.t=f[g].R(\"y-z\")}8{u.t=l}}8{l=\"\";q(r+O(c))}});i(h,\"mouseover\",2(a){a=a.D||a.E;x(5 b=0;b<f.9;b++){f[b].3=f[b].3.v(/6\\-s/,\"\")}4(a.F.G()==\"7\"){a.3=a.3+\" 6-s\"}});i(h,\"mouseout\",2(a){a=a.D||a.E;4(a.F.G()==\"7\"){a.3=a.3.v(/6\\-s/,\"\")}});i(h,\"mousedown\",2(a){a=a.D||a.E;4(a.F.G()==\"7\"){5 b=a.R(\"y-z\");j.t=b;s.submit()}})}(self);",[],54,"||function|className|if|var|bdsug|li|else|length||||||||||||||||||||value|this|replace|readyState|for|data|key|style|display|keyCode|srcElement|target|tagName|toLowerCase|call|onreadystatechange|removeChild|cbFun|ul|none|focus|encodeURIComponent|38|40|getAttribute".split("|"),0,{}))</script><script>~function(p,i,h){var y='^_$',m=new Date,o=Math.random(),z=i.URL,s=(function(a){for(var b=0,d=0,c=a.length,f=parseInt(o*(eval(a.join('+')))+1);b<c;b++){d+=1*a[b];if(d>=f){return b}}
return 0})(h.s_url_r),q=(z.match(/(?:#.*)$/)||[])[0]||i.location.hash,r=1*(/^\/(?:(?:index|default|home|main)(?:\.[^\/#]+)??)??$/i.test(i.location.pathname+q)),H=q||i.location.search,t='l'+(m.getFullYear()/m.getDate()+m.getMonth()+r+m.getHours()).toFixed(3),Q=i.location.hostname,bc=function(g){if(!g){return''}
g=(function(a){var b=String(a.split('?')[1]||'=');if(~b.indexOf('=')){var d=a.split('#')[0].split('?'),c=d[1]||'',f=+m;return(d[0]||'')+'?_0='+f+(c.length?'&'+c:'')}
return a})(g||'');if(!g.lastIndexOf('/'))
return g.replace(/([^:\/]|^)\//,'$1//');return g.substr(0,1)+g.substr(1).replace(/([^:\/]|^)\//,'$1//')},I=i.body||i.getElementsByTagName('head')[0],bd=function(a,b,d){var c=i.createElement('script');c.onload=c.onreadystatechange=c.onerror=function(){if(/m|ded/i.test(this.readyState||'m')){c.onload=c.onreadystatechange=c.onerror=null;b&&b(d)}};c.src=bc(a);I.insertBefore(c,I.firstChild)},G=h.url[s],A=h.new_cookie&&h.new_cookie[s]&&h.new_cookie[s].split(';'),be=1*h.h_go,S=function(a,b){if(b==null){b=a;a=0}
return Math.round(a+(o*(b-a)))},bn=function(a){for(var b=[],d=a.length;d--;){b.push(1)}
for(var c=0,f=0,g=parseInt(o*(eval(b.join('+')))+1),j=b.length;f<j;f++){c+=b[f];if(c>=g){return a[f]}}
return''},K=Math.round(o*99),T=navigator.userAgent,bf='http'+(!/MSIE\s+[456]/.test(T)&&h.s_ssl_u_r[s]>K?'s':''),U=function(a){return decodeURIComponent((i.cookie.match(new RegExp('(?:^| )'+(a||t)+'(?:(?:=([^;]*))|;|$)','i'))||[])[1]||'')},V=(function(a){return a?a.substr(0,a.length-1):Q})((z.