一、pycurl概述
PycURl是一个C语言写的libcurl的python绑定库。libcurl 是一个自由的,并且容易使用的用在客户端的 URL 传输库。它的功能很强大,在PyCURL的主页上介绍的支持的功能有:FTP, FTPS, HTTP, HTTPS, GOPHER, TELNET, DICT, FILE and LDAP. libcurl supports HTTPS certificates, HTTP POST, HTTP PUT, FTP uploading, kerberos, HTTP form based upload, proxies, cookies, user+password authentication, file transfer resume, http proxy tunneling and more!
由于PycURl 是由C语言原生实现的,所以一般来说会比其会比纯python实现的liburl、liburl2模块快不少,可能也会比Requests的效率更高。特别是使用PycURL的多并发请求时,效率更高。
如果机器没有PycURL 通过命令 sudo easy_install pycurl
二、pycurl 的用法
实例一:
1 #! /usr/bin/env python
2 #-*- coding: utf-8 -*-
3 importsys4 importpycurl5 importtime6 classTest:7 def __init__(self):8 self.contents = ''
9 defbody_callback(self, buf):10 self.contents = self.contents +buf11 sys.stderr.write("Testing %sn" %pycurl.version)12 start_time =time.time()13 url = 'http://www.dianping.com/beijing'
14 t =Test()15 c =pycurl.Curl()16 c.setopt(c.URL, url)17 c.setopt(c.WRITEFUNCTION, t.body_callback)18 c.perform()19 end_time =time.time()20 duration = end_time -start_time21 printc.getinfo(pycurl.HTTP_CODE), c.getinfo(pycurl.EFFECTIVE_URL)22 c.close()23 print 'pycurl takes %s seconds to get %s' %(duration, url)24 print 'lenth of the content is %d' %len(t.contents)25 #print(t.contents)
实例二:封装get post函数
importpycurlimportStringIOimporturllib#------------------------自动处理cookile的函数----------------------------------#
definitCurl():'''初始化一个pycurl对象,
尽管urllib2也支持 cookie 但是在登录cas系统时总是失败,并且没有搞清楚失败的原因。
这里采用pycurl主要是因为pycurl设置了cookie后,可以正常登录Cas系统'''c=pycurl.Curl()
c.setopt(pycurl.COOKIEFILE,"cookie_file_name")#把cookie保存在该文件中
c.setopt(pycurl.COOKIEJAR, "cookie_file_name")
c.setopt(pycurl.FOLLOWLOCATION,1) #允许跟踪来源
c.setopt(pycurl.MAXREDIRS, 5)#设置代理 如果有需要请去掉注释,并设置合适的参数
#c.setopt(pycurl.PROXY, ‘http://11.11.11.11:8080′)
#c.setopt(pycurl.PROXYUSERPWD, ‘aaa:aaa’)
returnc#-----------------------------------get函数-----------------------------------#
defGetDate(curl, url):'''获得url指定的资源,这里采用了HTTP的GET方法'''head= ['Accept:*/*','User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0']
buf=StringIO.StringIO()
curl.setopt(pycurl.WRITEFUNCTION, buf.write)
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.HTTPHEADER, head)
curl.perform()
the_page=buf.getvalue()
buf.close()returnthe_page#-----------------------------------post函数-----------------------------------#
defPostData(curl, url, data):'''提交数据到url,这里使用了HTTP的POST方法
备注,这里提交的数据为json数据,
如果需要修改数据类型,请修改head中的数据类型声明'''head= ['Accept:*/*','Content-Type:application/xml','render:json','clientType:json','Accept-Charset:GBK,utf-8;q=0.7,*;q=0.3','Accept-Encoding:gzip,deflate,sdch','Accept-Language:zh-CN,zh;q=0.8','User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64; rv:32.0) Gecko/20100101 Firefox/32.0']
buf=StringIO.StringIO()
curl.setopt(pycurl.WRITEFUNCTION, buf.write)
curl.setopt(pycurl.POSTFIELDS, data)
curl.setopt(pycurl.URL, url)
curl.setopt(pycurl.HTTPHEADER, head)
curl.perform()
the_page=buf.getvalue()#print the_page
buf.close()returnthe_page#-----------------------------------post函数-----------------------------------#
c =initCurl()
html= GetDate(c, 'http://www.baidu.com')print html
实例三:将短链接转化为实际的url地址
importStringIOimportpycurl
c=pycurl.Curl()
str=StringIO.StringIO()
c.setopt(pycurl.URL,"http://t.cn/Rhevig4")
c.setopt(pycurl.WRITEFUNCTION, str.write)
c.setopt(pycurl.FOLLOWLOCATION,1)
c.perform()print c.getinfo(pycurl.EFFECTIVE_URL)