Urllib2的使用，提供接口

Urllib2是用于获取URLs(统一资源定位符)的一个Python模块。它以urlopen函数的形式提供了非常简单的接口。能够使用各种不同的协议来获取网址。它还提供一个稍微复杂的接口用于处理常见的情况：如基本身份验证、cookies、proxies(代理)等。这些是由handlers和openers对象提供。
Urllib2使用相关的网络协议(FTP,http)，支持多种获取URLs的方案(以URL前面的”: ”定义,如：ftp://python.org)，这里主要讲最常见的http。

import urllib2
response = urllib2.urlopen('http://baidu.com')
html = response.read()
print html

res = urllib2.Request(url, headers=i_headers,data = json.dumps(body))
urop = urllib2.urlopen(res,timeout = 3)
rep = urop.read()

import urllib2
req = urllib2.Request('http://python.org')
response = urllib2.urlopen(req)
the_page = response.read()
print the_page

#有时你想发送一个数据到URL(通常这个URL指向到CGI(公共网关接口)脚本或其他Web应用程序)。
#在HTTP中，经常使用POST请求。这个通常是浏览器做的，当你在你填写的网站上提交HTML表单。
#不是所有的POSTs都使用表单的形式：你可以使用POST传输任意数据到自己的应用中。在通常的HTML表单中，这些数据需要以标准的形式进行编码，然后传递到Request对象作为data参数。编码是通过使用urllib库而不是urllib2库。
    import urllib
    import urllib2
    url = 'http://www.someserver.com/cgi-bin/register.cgi'
    values = {}
    values['name'] = 'Michael Foord'
    values['location'] = 'Northampton'
    values['language'] = 'Python'

    data = urllib.urlencode(values) #数据进行编码
    req = urllib2.Request(url,data) #作为data参数传递到Request对象中
    response = urllib2.urlopen(req)
    the_page = response.read()
    print the_page
#如果你不传递data参数，urllib2使用GET请求。GET和POST请求不同的是，POST请求经常有副作用：它们在某种程度上改变系统的状态。
#尽管HTTP标准有明确的说明POSTs总是引起副作用，而GET请求从不会引起副作用，没有什么可以防止GET请求没有副作用，POST请求有副作用。

#数据也可以通过在URL本身中HTTP GET请求来编码。
    >>> import urllib2
    >>> import urllib
    >>> data = {}
    >>> data['name'] = 'Somebody Here'
    >>> data['location'] = 'Northampton'
    >>> data['language'] = 'Python'
    >>> url_values = urllib.urlencode(data)
    >>> print url_values
    name=Somebody+Here&language=Python&location=Northampton
    >>> url = 'http://www.example.com/example.cgi'
    >>> full_url = url + '?' + url_values
    >>> data = urllib2.urlopen(full_url) #这个完整的URL是添加一个？到URL中，其次是编码的值。

在这里讨论一个特定的HTTP头，说明如何添加头到你的HTTP请求。
一些网站(google)不喜欢由程序来访问或不同的浏览器发送不同的版本，默认情况下，urllib2标识自己为python urllib / x.y（x.y是Python版本号eg : 2.7）,这个可能会混淆网站，或只是简单的不工作。
浏览器标识自己的方式是通过用户代理（User-Agent）头。当创建一个Request对象时，你可以通过一个Headers的字典。下面的例子使相同的请求，但将其标识为一个版本的Internet Explorer。
    import urllib
    import urllib2
    url = 'http://www.someserver.com/cgi-bin/register.cgi'
    user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'

    values = {}
    values['name'] = 'Michael Foord'
    values['location'] = 'Northampton'
    values['language'] = 'Python'

    headers = { 'User-Agent' : user_agent }
    data = urllib.urlencode(values)
    req = urllib2.Request(url,data,headers)
    response = urllib2.urlopen(req)
    the_page = response.read()
    print the_page
response还有两个常用的方法(info、geturl)。

返回错误

responses = {

    100: ('Continue', 'Request received, please continue'),
    101: ('Switching Protocols',
    'Switching to new protocol; obey Upgrade header'),

    200: ('OK', 'Request fulfilled, document follows'),
    201: ('Created', 'Document created, URL follows'),
    202: ('Accepted',
    'Request accepted, processing continues off-line'),
    203: ('Non-Authoritative Information', 'Request fulfilled from cache'),
    204: ('No Content', 'Request fulfilled, nothing follows'),
    205: ('Reset Content', 'Clear input form for further input.'),
    206: ('Partial Content', 'Partial content follows.'),

    300: ('Multiple Choices',
    'Object has several resources -- see URI list'),
    301: ('Moved Permanently', 'Object moved permanently -- see URI list'),
    302: ('Found', 'Object moved temporarily -- see URI list'),
    303: ('See Other', 'Object moved -- see Method and URL list'),
    304: ('Not Modified',
    'Document has not changed since given time'),
    305: ('Use Proxy',
    'You must use proxy specified in Location to access this '
    'resource.'),
    307: ('Temporary Redirect',
    'Object moved temporarily -- see URI list'),

    400: ('Bad Request',
    'Bad request syntax or unsupported method'),
    401: ('Unauthorized',
    'No permission -- see authorization schemes'),
    402: ('Payment Required',
    'No payment -- see charging schemes'),
    403: ('Forbidden',
    'Request forbidden -- authorization will not help'),
    404: ('Not Found', 'Nothing matches the given URI'),
    405: ('Method Not Allowed',
    'Specified method is invalid for this server.'),
    406: ('Not Acceptable', 'URI not available in preferred format.'),
    407: ('Proxy Authentication Required', 'You must authenticate with '
    'this proxy before proceeding.'),
    408: ('Request Timeout', 'Request timed out; try again later.'),
    409: ('Conflict', 'Request conflict.'),
    410: ('Gone',
    'URI no longer exists and has been permanently removed.'),
    411: ('Length Required', 'Client must specify Content-Length.'),
    412: ('Precondition Failed', 'Precondition in headers is false.'),
    413: ('Request Entity Too Large', 'Entity is too large.'),
    414: ('Request-URI Too Long', 'URI is too long.'),
    415: ('Unsupported Media Type', 'Entity body in unsupported format.'),
    416: ('Requested Range Not Satisfiable',
    'Cannot satisfy request range.'),
    417: ('Expectation Failed',
    'Expect condition could not be satisfied.'),

    500: ('Internal Server Error', 'Server got itself in trouble'),
    501: ('Not Implemented',
    'Server does not support this operation'),
    502: ('Bad Gateway', 'Invalid responses from another server/proxy.'),
    503: ('Service Unavailable',
    'The server cannot process the request due to a high load'),
    504: ('Gateway Timeout',
    'The gateway server did not receive a timely response'),
    505: ('HTTP Version Not Supported', 'Cannot fulfill request.'),
}

当urlopen无法处理响应时将引起URLError，通常，网络没有连接或没有路由到指定的服务器或指定的服务器不存在将引起URLError.在这种情形下，异常引发将有个reason属性，这是一个包含错误代码和一个文本错误信息的元组。

    import urllib2
    req = urllib2.Request('http://www.pretend_server.org')
    try:
        urllib2.urlopen(req)
    except urllib2.URLError,e:
        print e.reason

如果你想编写HTTPError或URLError有两个基本方法。(更喜欢第二种方法)
Number1
同时处理HTTPError和URLError，应该把HTTPError放在URLError前面。由于HTTPError是URLError的子类，否则URLError也会捕获一个HTTPError错误。

from urllib2 import Request,urlopen,URLError,HTTPError

req = Request('http://www.python.org/fish.html')
try:
    response = urlopen(req)
except HTTPError,e:
    print 'The server couldn\'t fulfill the request'
    print 'Error code:',e.code
except URLError,e:
    print 'We failed to reache a server.'
    print 'Reason:',e.reason
else:
    #everything is fine
    response.getcode()

由于HTTPError是URLError的子类，可以避免导入HTTPError.改进如下:

    from urllib2 import Request,urlopen,URLError

    req = Request('http://www.python.org/fish.html')
    try:
        response = urlopen(req)
    except URLError,e:
        if hasattr(e,'reason'):
            print 'We failed to reach a server.'
            print 'Reason:',e.reason
        elif hasattr(e,'code'):
            print 'The server couldn\'t fulfill the request'
            print 'Error code:',e.code
        else:
            #everything is fine
            response.getcode()

注意，URLError是内置异常IOError的子类，同样也可避免导入URLError.在某些情况下，urllib2也可能会引起scoker.error。

from urllib2 import Request,urlopen

    req = Request('http://www.python.org/fish.html')
    try:
        response = urlopen(req)
    except IOError,e:
        if hasattr(e,'reason'):
            print 'We failed to reach a server.'
            print 'Reason:',e.reason
        elif hasattr(e,'code'):
            print 'The server couldn\'t fulfill the request'
            print 'Error code:',e.code
    else:
        #everything is fine
        response.getcode()

BadStatusLine and HttpException
由一两种情况例外，不继承IOError引起的异常，一种是由httplib模块定义的BadStatusLine异常，当请求的页页面是空白时会引起这个异常。它没有继承IOError，而是从HttpException继承(在httplib中重新定义，直接从Exception中继承)，可能还有其他的情况下，这些例外可以渗入到用户urllib2。你可以将这些异常类型，从httplib中直接抓取，或者用一个"catch-all"异常语句来处理任何发生的错误。
参考文章