python_urllib2下载网页的三种方式

最新推荐文章于 2020-11-30 08:10:37 发布

杨MAX洁

最新推荐文章于 2020-11-30 08:10:37 发布

阅读量797

点赞数

分类专栏： Python

本文链接：https://blog.csdn.net/Hpu_A/article/details/51008535

版权

Python 专栏收录该内容

30 篇文章 1 订阅

订阅专栏

# -*- coding: utf-8 -*-
import urllib2
import  cookielib
url="http://www.baidu.com" #define URL

#  three ways to capturing  webs
print "first way:"
response1 = urllib2.urlopen(url)
print response1.getcode()#return 200,the web can be accessible
print  len(response1.read())# return the length of this web

print "second way:"
request = urllib2.Request(url)#调用request对象
request.add_header("user-agent" , "Mozilla/5.0")#把爬虫伪装成一个浏览器，Mozilla/5.0是火狐客户端浏览器版本

response2 = urllib2.urlopen(request)
print response2.getcode()
print  len(response2.read())

print "third way:"
cj= cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)

response3 = urllib2.urlopen(url)
print response3.getcode()
print  cj #打印cookie内容
print response3.read()