12.常用模块

datetime
collections:namedtuple,deque,defaultdict,OrderedDict,Counter
base64
hashlib,md5
itertools:无限迭代器,takewhile,cycle,repeat,分组
contextlib
urllib
XML:sax,dom

1.datetime
当前时间

from datetime import datetime

now = datetime.now()
print(now, type(now))  # 当前时间
---
2016-11-08 20:54:09.304088 <class 'datetime.datetime'>

在计算机中，时间是以一个long值表示的，也可以叫时间戳，1970年1月1日 00:00:00 UTC+00:00时区的时刻称为epoch time，记为0 ，1970年以前的时间戳为负数，当前时间就是相对于epoch time的秒数，在Python中叫timestamp。

timestamp = 0 = 1970-1-1 00:00:00 UTC+0:00

对应的北京时间是：

timestamp = 0 = 1970-1-1 08:00:00 UTC+8:00

timestamp的值与时区毫无关系，timestamp一旦确定，UTC时间就会确定，全球各地的时间在同一时刻的timestamp都是一样的。

datetime转timestamp:

d = datetime(now.year, now.month, now.day, now.hour, now.minute, now.second)
print(d.timestamp())
---
1478610428.0

后面的小数代表毫秒，java的System.currentTimeMillions()获取的值除以1000就等于Python的这个timestamp。

timestamp转datetime

print(datetime.fromtimestamp(timestamp)) 
---
2016-11-08 21:07:08

上面的例子是让timestamp与本地时间做转换，本地时间是指当前操作系统设定的时区的时间，例如我本地设定的是东8区的北京时间，上面转换的实际上就是UTC+8:00时区的时间

2016-11-08 21:07:08 UTC+8:00

此刻格林威治标准时间与北京相差8小时，也就是UTC+0:00时区时间：

2016-11-08 13:07:08 UTC+0:00

如果想把timestamp直接转为标准时间：

print(datetime.utcfromtimestamp(timestamp))  # 转为utc的时间

字符串类型的时间转datetime:

strptime = datetime.strptime('2016-11-08 21:07:08', '%Y-%m-%d %H:%M:%S')
print(strptime, type(strptime))
---
2016-11-08 21:07:08 <class 'datetime.datetime'>

datetime转字符串类型：

print(strptime.strftime('%Y.%m.%d %H:%M---%a'))
---
2016.11.08 21:07---Tue

时间加减：

from datetime import datetime, timedelta
strptime = datetime.strptime('2016-11-08 21:07:08', '%Y-%m-%d %H:%M:%S')
print(strptime, type(strptime))


print(strptime + timedelta(days=1))
print(strptime + timedelta(hours=3))
print(strptime + timedelta(days=1, hours=3))
print(strptime - timedelta(days=1, hours=3))
---
2016-11-08 21:07:08 <class 'datetime.datetime'>
2016-11-09 21:07:08
2016-11-09 00:07:08
2016-11-10 00:07:08
2016-11-07 18:07:08

把本地时间转换为utc时间（本地时间）：

# 转换为utc时间
t = timezone(timedelta(hours=8))
now = datetime.now()
print(now)
utc_dt = now.replace(tzinfo=t)  # 直接设置时区
print(utc_dt)
---
2016-11-08 22:00:31.905879
2016-11-08 22:00:31.905879+08:00

转换为utc时间后就可以通过时区转换得到其它时区的时间了：

utc_time = datetime.utcnow().replace(tzinfo=timezone.utc)
astimezone_bj = utc_time.astimezone(timezone(timedelta(hours=8)))  # 北京时间
astimezone_dj = utc_time.astimezone(timezone(timedelta(hours=9)))  # 东京时间
astimezone_bj2dj = astimezone_bj.astimezone(timezone(timedelta(hours=9)))  # 北京时间转东京时区时间
print(astimezone_bj)
print(astimezone_dj)
print(astimezone_bj2dj)
---
2016-11-08 22:04:23.913149+08:00
2016-11-08 23:04:23.913149+09:00
2016-11-08 23:04:23.913149+09:00

主要思路就是先拿到当前的datetime，但要知道当前正确的时区，然后强制加上时区属性，作为基准时间，因为datetime的tzinfo属性默认为None，然后用带时区的datetime通过astimezone()方法就可以转换到任意时区。（并不一定要用UTC+0:00时区的时间转，任意带时区的时间都能转，例如上面示例中的北京时区的时间转东京时间）

2.collections
2.1namedtuple

from collections import namedtuple

Point = namedtuple('Point', ['x', 'y'])
p = Point(1, 2)
print(p.x, p.y)
---
1 2

可理解为定义一个tuple对象，名为Point，元素个数是固定的，而且属性不是用索引来引用，而是用定义的x，y引用。

2.2deque

d = deque([1,2,3])
d.append(4)
d.appendleft(0)

print(d)
---
deque([0, 1, 2, 3, 4])

list查询快，增删慢，deque是为了高效实现增删操作的双向列表。

2.3.defaultdict

defaultd = defaultdict(lambda :"sss")
defaultd['key1'] = 'abc'
print(defaultd['key1'])
print(defaultd['key2'])
---
abc
sss

使用dict时如果获取的key不存在，会报keyError，用defaultdict可以设置一个默认值，获取的key不存在时就会返回这个默认值而不会报错。

2.4OrderedDict

# OrderedDict
d1 = dict([('x', 1), ('y', 2), ('z', 3)])  # 无序
print(d1)

d2 = OrderedDict([('x', 1), ('y', 2), ('z', 3)])#有序
for i in d2:
    print('%s=%s' % (i, d2[i]))
---
{'z': 3, 'x': 1, 'y': 2}
x=1
y=2
z=3

2.5Counter

# Counter
c = Counter()
for i in 'aaabcccdd':
    c[i] = c[i] + 1
print(c)
---
Counter({'a': 3, 'c': 3, 'd': 2, 'b': 1})

计算这个字符串里每个字符出现的字数。

3.base64
base64是一种以64个字符来表示任意二进制数据的方法；

import base64

print(base64.b64encode(b'abc'))
print(base64.b64decode(b'YWJj'))
---
b'YWJj'
b'abc'

由于标准的Base64编码后的字符中可能出现“+”和“/”，这两符号在url中不能直接作为参数，所以有一种url safe的base64编码，其实就是把字符+和/分别变为-和_：

print(base64.b64encode(b'i\xb7\x1d\xfb\xef\xff'))  # 普通编码，会出现+和/
print(base64.urlsafe_b64encode(b'i\xb7\x1d\xfb\xef\xff'))  # url safe编码，把+和/变成了-_
print(base64.urlsafe_b64decode(b'abcd--__'))
print(base64.urlsafe_b64decode(b'abcd++//'))
---
b'abcd++//'
b'abcd--__'
b'i\xb7\x1d\xfb\xef\xff'
b'i\xb7\x1d\xfb\xef\xff'

由于Base64编码后的字符里可能有“=”，但是=在URL或cookie里会造成歧义，所以很多编码后会把=去掉，去掉=的字符解码方式要做一些处理，因为Base64是把3个字节变成4个字节，所以编码后的字符长度永远是4的倍数，所以在解码前先把要解码的字符加上=，把长度变成4的倍数就可以正常解码了：

def safe_base64_decode(c):
    clength = 0
    if isinstance(c, bytes):
        c = str(c, encoding='utf-8')
        clength = len(c) % 4
    return base64.b64decode(c + '=' * clength)


assert b'abcd' == safe_base64_decode(b'YWJjZA=='), safe_base64_decode('YWJjZA==')
assert b'abcd' == safe_base64_decode(b'YWJjZA'), safe_base64_decode('YWJjZA')

4.hashlib
md5算法：

import hashlib

md5 = hashlib.md5()
md5.update("testtest".encode('utf-8'))
print(md5.hexdigest())

md5 = hashlib.md5()
md5.update("test".encode('utf-8'))
md5.update("test".encode('utf-8'))
print(md5.hexdigest())
---
05a671c66aefea124cc08b76ea6d30bb
05a671c66aefea124cc08b76ea6d30bb

update可多次调用，结果是一样的；

5.itertools
5.1无限迭代器:

import itertools

count = itertools.count(10, 2)  # 从10开始，步长2
for i in count:
    print(i)
    if i >= 20:
        break
---
10
12
14
16
18
20

5.2用Python提供的takewhile方法也可以约束：

count1 = itertools.count(10,2)
takewhile = itertools.takewhile(lambda x: x <= 20, count1)
print(list(takewhile))
---
[10, 12, 14, 16, 18, 20]

5.3 cycle

cycle = itertools.cycle("123")
n = 0
for i in cycle:
    n = n+1
    print(i)
    if n > 5:
        break
---
1
2
3
1
2
3

5.4 repeat

repeat = itertools.repeat('TEST', 3)  # 输出TEST,共输出3次
for i in repeat:
    print(i)
---
TEST
TEST
TEST

5.5把两组迭代对象串起来：

chain = itertools.chain('ABC', 'DEF')
for i in chain:
    print(i)
---
A
B
C
D
E
F

5.6 分组：

groupby = itertools.groupby('AAABBC')
for key,group in groupby:
    print(key,list(group))
---
A ['A', 'A', 'A']
B ['B', 'B']
C ['C']

分组是通过内置函数完成的，只要返回的元素值相等，就被认为是同一组的，函数的返回值作为组的Key，如果要忽略大小写，可让大小和小写返回相同的key，把大写变小写或小写变大写：

groupby = itertools.groupby('AAaBBC', lambda c: c.upper())
for key, group in groupby:
    print(key, list(group))
---
A ['A', 'A', 'a']
B ['B', 'B']
C ['C']

6.contextlib
在操作文件资源时，用完后要释放资源，释放资源的一个方法是使用try。。。finally，在finally里释放资源；
另一种简单的方法是使用with语句：

with open('test12_6_contextlib.py','r') as f:
    print(f.read(30))

但是只有实现了上下文管理的对象才能使用with语句，实现上下文管理有如下几种方式：
1.实现enter和exit方法：

class TestContext:
    def __init__(self, name):
        self.name = name

    def __enter__(self):
        print('before')
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        if exc_type:
            print('Error')
        else:
            print('End')

    def getName(self):
        print("name is %s" % self.name)


with TestContext('enter_exit') as t:
    t.getName()
---
before
name is enter_exit
End

2.使用contextlib库的@contextmanager

from contextlib import contextmanager
# contextmanager
class TestContext1:
    def __init__(self, name):
        self.name = name

    def getName(self):
        print("TestContext1 name is %s" % self.name)


@contextmanager
def getTestContext1Name(name):
    print('before')
    t = TestContext1(name)
    yield t
    print('end')


with getTestContext1Name('contextmanager') as t1:
    t1.getName()
---
before
TestContext1 name is contextmanager
end

给一个字符串前后加标签：

@contextmanager
def htmTag(tagName):
    print("<%s>" % tagName)
    yield
    print("</%s>" % tagName)


with htmTag("div"):
    print('hello')
    print('python!')
---
<div>
hello
python!
</div>

yield代表执行with语句内部的所有代码；

with语句操作文件实际是这样的：

@contextmanager
def myopen(filename, mode='r'):
    f = open(filename, mode)
    try:
        yield f
    finally:
        f.close()


with myopen('test12_6_contextlib.py', 'r') as f:
    print('myopen', f.name)
---
myopen test12_6_contextlib.py

3.使用@closing把任意对象变成上下文对象，以可以使用with语句

from urllib.request import urlopen
with closing(urlopen('https://www.python.org')) as page:
    for i in page:
        print(i)

closing是一个被@contextmanager装饰过的generator：

@contextmanager
def closing(method):
    try:
        yield method
    finally:
        method.close()

7.urllib
7.1 get请求

from urllib import request, parse

with request.urlopen('http://www.sojson.com/open/api/weather/json.shtml?city=%s' % parse.quote("北京")) as f:
    data = f.read()
    print('Status:', f.status, f.reason)
    for k, v in f.getheaders():
        print('%s: %s' % (k, v))
    print('response:', data.decode('utf-8'))
---
Status: 200 OK
Server: nginx
Date: Sat, 18 Nov 2017 13:34:33 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: close
Content-Disposition: inline;filename=f.txt
response: {"date":"20171118","message":"Success !","status":200,"city":"北京","count":7,"data":{"shidu":"16%","pm25":6.0,"pm10":27.0,"quality":"优","wendu":"-1","ganmao":"各类人群可自由活动","yesterday":{"date":"17日星期五","sunrise":"07:00","high":"高温 6.0℃","low":"低温 -4.0℃","sunset":"16:58","aqi":34.0,"fx":"北风","fl":"5-6级","type":"多云","notice":"悠悠的云里有淡淡的诗"},"forecast":[{"date":"18日星期六","sunrise":"07:01","high":"高温 5.0℃","low":"低温 -6.0℃","sunset":"16:57","aqi":56.0,"fx":"西南风","fl":"3-4级","type":"晴","notice":"晴空万里，去沐浴阳光吧"},{"date":"19日星期日","sunrise":"07:02","high":"高温 7.0℃","low":"低温 -5.0℃","sunset":"16:56","aqi":135.0,"fx":"东南风","fl":"<3级","type":"多云","notice":"绵绵的云朵，形状千变万化"},{"date":"20日星期一","sunrise":"07:03","high":"高温 10.0℃","low":"低温 -3.0℃","sunset":"16:55","aqi":109.0,"fx":"西南风","fl":"<3级","type":"晴","notice":"天气干燥，请适当增加室内湿度"},{"date":"21日星期二","sunrise":"07:04","high":"高温 9.0℃","low":"低温 -2.0℃","sunset":"16:55","aqi":201.0,"fx":"西风","fl":"<3级","type":"多云","notice":"绵绵的云朵，形状千变万化"},{"date":"22日星期三","sunrise":"07:06","high":"高温 8.0℃","low":"低温 -2.0℃","sunset":"16:54","aqi":57.0,"fx":"西北风","fl":"4-5级","type":"晴","notice":"晴空万里，去沐浴阳光吧"}]}}

设置header参数：

request_request = request.Request('https://www.baidu.com/')
request_request.add_header('User-Agent', 'Test')  # 给请求增加header参数
request_request.add_header('token', '123456')
with request.urlopen(request_request) as f:
    print('Status:', f.status, f.reason)
    for k, v in f.getheaders():
        print('%s: %s' % (k, v))
    print('response:', f.read().decode('utf-8'))

post请求：

params = parse.urlencode([
    ('key1', 'value1'),
    ('key2', 'value2')])
req_post = request.Request('http://www.baidu.com')
with request.urlopen(req_post,data=params.encode('utf-8')) as f:
    print('Status:', f.status, f.reason)
for k, v in f.getheaders():
    print('%s: %s' % (k, v))
print('Data:', f.read().decode('utf-8'))

8.XML
8.1 Sax解析：

from pyexpat import ParserCreate

xml = r'''
<head>
    <meta charset="utf-8" />
    <title>淘宝网 - 淘！我喜欢</title>
    <meta name="spm-id" content="a21bo" />
</head>
'''


class MySaxHandler(object):
    def start_element(self, name, attrs):
        print('start element:%s,attr:%s' % (name, str(attrs)))

    def end_element(self, name):
        print('sax:end element:%s' % name)

    def char_data(self, content):
        print('sax:content:%s' % content)


handler = MySaxHandler()
parser = ParserCreate()
parser.StartElementHandler = handler.start_element
parser.EndElementHandler = handler.end_element
parser.CharacterDataHandler = handler.char_data
parser.Parse(xml)
---
start element:head,attr:{}
sax:content:

sax:content:    
start element:meta,attr:{'charset': 'utf-8'}
sax:end element:meta
sax:content:

sax:content:    
start element:title,attr:{}
sax:content:淘宝网 - 淘！我喜欢
sax:end element:title
sax:content:

sax:content:    
start element:meta,attr:{'name': 'spm-id', 'content': 'a21bo'}
sax:end element:meta
sax:content:

sax:end element:head

8.2Dom解析

dom = minidom.parse('test.xml')
rootElement = dom.documentElement
print('名称：%s,值：%s,类型：%s' % (rootElement.nodeName, rootElement.nodeValue, rootElement.nodeType))
print(rootElement.childNodes[0].nodeName)
list_ = rootElement.getElementsByTagName('list')
for li in list_:
    print('------')
    print(li.getAttribute('id'))
    for node in li.childNodes:
        if node.nodeType == 1:
            print(node.nodeName + ":" + node.childNodes[0].data)

---
名称：info,值：None,类型：1
intro
------
001
name:book1
price:100
------
002
name:book2
price:200

test.xml:

<?xml version="1.0" encoding="utf-8"?>
<info><intro>Book message</intro>
    <list id='001'>
        <name>book1</name>
        <price>100</price>
    </list>

    <list id='002'>
        <name>book2</name>
        <price>200</price>
    </list>
</info>