模块，用一砣代码实现了某个功能的代码集合。
类似于面向过程编程，函数式编程则完成一个功能，其他代码用来调用即可，提供了代码的重用性和代码间的耦合。而对于一个复杂的功能来，可能需要多个函数才能完成（函数又可以在不同的.py文件中），n个 .py 文件组成的代码集合就称为模块。
如：os 是系统相关的模块；file是文件操作相关的模块
模块分为三种：
- 自定义模块
- 第三方模块
- 内置模块
自定义模块
定义模块
导入模块
Python之所以应用越来越广泛，在一定程度上也依赖于其为程序员提供了大量的模块以供使用，如果想要使用模块，则需要导入。导入模块有一下几种方法：
import module
from module.xx.xx import xx
from module.xx.xx import xx as rename
from module.xx.xx import * 导入模块其实就是告诉Python解释器去解释那个py文件
- 导入一个py文件，解释器解释该py文件
- 导入一个包，解释器解释该包下的 __init__.py 文件
那么问题来了，导入模块时是根据那个路径作为基准来进行的呢？即：sys.path
['D:\\PythonProject\\0313', 'D:\\PythonProject', 'D:\\Program Files\\Python3\\python35.zip', 'D:\\Program Files\\Python3\\DLLs', 'D:\\Program Files\\Python3\\lib', 'D:\\Program Files\\Python3', 'D:\\Program Files\\Python3\\lib\\site-packages']
this is a0312/login/bak
内置模块
内置模块是Python自带的功能，在使用内置模块相应的功能时，需要【先导入】再【使用】
sys模块
1 sys.argv 命令行参数List，第一个元素是程序本身路径 2 sys.exit(n) 退出程序，正常退出时exit(0) 3 sys.version 获取Python解释程序的版本信息 4 sys.maxint 最大的Int值 5 sys.path 返回模块的搜索路径，初始化时使用PYTHONPATH环境变量的值 6 sys.platform 返回操作系统平台名称 7 sys.stdin 输入相关 8 sys.stdout 输出相关 9 sys.stderror 错误相关
os模块
os.getcwd() 获取当前工作目录，即当前python脚本工作的目录路径 os.chdir("dirname") 改变当前脚本工作目录；相当于shell下cd os.curdir 返回当前目录: ('.') os.pardir 获取当前目录的父目录字符串名：('..') os.makedirs('dir1/dir2') 可生成多层递归目录 os.removedirs('dirname1') 若目录为空，则删除，并递归到上一级目录，如若也为空，则删除，依此类推 os.mkdir('dirname') 生成单级目录；相当于shell中mkdir dirname os.rmdir('dirname') 删除单级空目录，若目录不为空则无法删除，报错；相当于shell中rmdir dirname os.listdir('dirname') 列出指定目录下的所有文件和子目录，包括隐藏文件，并以列表方式打印 os.remove() 删除一个文件 os.rename("oldname","new") 重命名文件/目录 os.stat('path/filename') 获取文件/目录信息 os.sep 操作系统特定的路径分隔符，win下为"\\",Linux下为"/" os.linesep 当前平台使用的行终止符，win下为"\t\n",Linux下为"\n" os.pathsep 用于分割文件路径的字符串 os.name 字符串指示当前使用平台。win->'nt'; Linux->'posix' os.system("bash command") 运行shell命令，直接显示 a=os.popen("bash command").read() 不直接显示，print a可以打印 os.environ 获取系统环境变量 os.path.abspath(path) 返回path规范化的绝对路径 os.path.split(path) 将path分割成目录和文件名二元组返回 os.path.dirname(path) 返回path的目录。其实就是os.path.split(path)的第一个元素 os.path.basename(path) 返回path最后的文件名。如何path以／或\结尾，那么就会返回空值。即os.path.split(path)的第二个元素 os.path.exists(path) 如果path存在，返回True；如果path不存在，返回False os.path.isabs(path) 如果path是绝对路径，返回True os.path.isfile(path) 如果path是一个存在的文件，返回True。否则返回False os.path.isdir(path) 如果path是一个存在的目录，则返回True。否则返回False os.path.join(path1[, path2[, ...]]) 将多个路径组合后返回，第一个绝对路径之前的参数将被忽略 os.path.getatime(path) 返回path所指向的文件或者目录的最后存取时间 os.path.getmtime(path) 返回path所指向的文件或者目录的最后修改时间
datetime模块
参考 https://www.cnblogs.com/snow-backup/p/5063665.html
#把datetime转成字符串
def datetime_toString(dt):
return dt.strftime("%Y-%m-%d-%H")
#把字符串转成datetime
def string_toDatetime(string):
return datetime.strptime(string, "%Y-%m-%d-%H")
#把字符串转成时间戳形式
def string_toTimestamp(strTime):
return time.mktime(string_toDatetime(strTime).timetuple())
#把时间戳转成字符串形式
def timestamp_toString(stamp):
return time.strftime("%Y-%m-%d-%H", tiem.localtime(stamp))
#把datetime类型转外时间戳形式
def datetime_toTimestamp(dateTim):
return time.mktime(dateTim.timetuple())shutil模块
高级的 文件、文件夹、压缩包 处理模块
将文件内容拷贝到另一个文件中
import shutil
shutil.copyfileobj(open('old.xml','r'), open('new.xml', 'w'))shutil.copyfile(src, dst)
拷贝文件
shutil.copyfile('f1.log', 'f2.log')
shutil.copymode(src, dst)
仅拷贝权限。内容、组、用户均不变
shutil.copymode('f1.log', 'f2.log')
shutil.copystat(src, dst)
仅拷贝状态的信息，包括：mode bits, atime, mtime, flags
shutil.copystat('f1.log', 'f2.log')
shutil.copy(src, dst)
拷贝文件和权限
import shutil
shutil.copy('f1.log', 'f2.log')shutil.copy2(src, dst)
拷贝文件和状态信息
import shutil
shutil.copy2('f1.log', 'f2.log')
shutil.ignore_patterns(*patterns)
shutil.copytree(src, dst, symlinks=False, ignore=None)
递归的去拷贝文件夹
import shutil
shutil.copytree('folder1', 'folder2', ignore=shutil.ignore_patterns('*.pyc', 'tmp*'))import shutil
shutil.copytree('f1', 'f2', symlinks=True, ignore=shutil.ignore_patterns('*.pyc', 'tmp*'))
shutil.rmtree(path[, ignore_errors[, onerror]])
递归的去删除文件
import shutil
shutil.rmtree('folder1')
shutil.move(src, dst)
递归的去移动文件，它类似mv命令，其实就是重命名。
import shutil
shutil.move('folder1', 'folder3')shutil.make_archive(base_name, format,...)
创建压缩包并返回文件路径，例如：zip、tar
创建压缩包并返回文件路径，例如：zip、tar
- base_name： 压缩包的文件名，也可以是压缩包的路径。只是文件名时，则保存至当前目录，否则保存至指定路径，
如：www =>保存至当前路径
如：/Users/wupeiqi/www =>保存至/Users/wupeiqi/ - format： 压缩包种类，“zip”, “tar”, “bztar”，“gztar”
- root_dir： 要压缩的文件夹路径（默认当前目录）
- owner： 用户，默认当前用户
- group： 组，默认当前组
- logger： 用于记录日志，通常是logging.Logger对象
#将 /Users/wupeiqi/Downloads/test 下的文件打包放置当前程序目录
import shutil
ret = shutil.make_archive("wwwwwwwwww", 'gztar', root_dir='/Users/wupeiqi/Downloads/test')
#将 /Users/wupeiqi/Downloads/test 下的文件打包放置 /Users/wupeiqi/目录
import shutil
ret = shutil.make_archive("/Users/wupeiqi/wwwwwwwwww", 'gztar', root_dir='/Users/wupeiqi/Downloads/test')
shutil 对压缩包的处理是调用 ZipFile 和 TarFile 两个模块来进行的，详细：
import zipfile # 压缩 z = zipfile.ZipFile('laxi.zip', 'w') z.write('a.log') z.write('data.data') z.close() # 解压 z = zipfile.ZipFile('laxi.zip', 'r') z.extractall() z.close()
import tarfile # 压缩 tar = tarfile.open('your.tar','w') tar.add('/Users/wupeiqi/PycharmProjects/bbs2.log', arcname='bbs2.log') tar.add('/Users/wupeiqi/PycharmProjects/cmdb.log', arcname='cmdb.log') tar.close() # 解压 tar = tarfile.open('your.tar','r') tar.extractall() # 可设置解压地址 tar.close()
json & pickle 模块
序列化：把内存的数据保存到文件
用于序列化的两个模块
- json，用于字符串 和 python数据类型间进行转换。优点是所有语言通用，缺点是只能序列化基本的数据类型(函数，类，日期等格式不支持)
- pickle，用于python特有的类型 和 python的数据类型间进行转换。优点是可以序列化python的所有数据类型，缺点是只能用于 python语言
Json模块提供了四个功能：dumps、dump、loads、load
pickle模块提供了四个功能：dumps、dump、loads、load
shelve 模块
目的：代替pickle
shelve模块是一个简单的k,v将内存数据通过文件持久化的模块，可以持久化任何pickle可支持的python数据格式(对pickle再做了一次封装，目的是为了让调用更简单)
pickle的缺点：load和dump的顺序一致,不能重复load
shelve的优点：1.可以按key get;2.可以重复get
import shelve
d = shelve.open('shelve_test') # 打开一个文件
class Test(object):
def __init__(self, n):
self.n = n
t = Test(123)
t2 = Test(123334)
name = ["alex", "rain", "test"]
d["test"] = name # 持久化列表
d["t1"] = t # 持久化类
d["t2"] = t2
d.close()
import shelve
a = shelve.open('shelve_test') # 打开一个文件
b = a.get("t2")xml处理模块
xml是实现不同语言或程序之间进行数据交换的协议，跟json差不多，但json使用起来更简单，不过，古时候，在json还没诞生的黑暗年代，大家只能选择用xml呀，至今很多传统公司如金融行业的很多系统的接口还主要是xml。
xml的格式如下，就是通过<>节点来区别数据结构的:
1 <?xml version="1.0"?> 2 <data> 3 <country name="Liechtenstein"> 4 <rank updated="yes">2</rank> 5 <year>2008</year> 6 <gdppc>141100</gdppc> 7 <neighbor name="Austria" direction="E"/> 8 <neighbor name="Switzerland" direction="W"/> 9 </country> 10 <country name="Singapore"> 11 <rank updated="yes">5</rank> 12 <year>2011</year> 13 <gdppc>59900</gdppc> 14 <neighbor name="Malaysia" direction="N"/> 15 </country> 16 <country name="Panama"> 17 <rank updated="yes">69</rank> 18 <year>2011</year> 19 <gdppc>13600</gdppc> 20 <neighbor name="Costa Rica" direction="W"/> 21 <neighbor name="Colombia" direction="E"/> 22 </country> 23 </data> 24
xml协议在各个语言里的都 是支持的，在python中可以用以下模块操作xml
1 import xml.etree.ElementTree as ET 2 3 tree = ET.parse("xmltest.xml") 4 root = tree.getroot() 5 print(root.tag) 6 7 #遍历xml文档 8 for child in root: 9 print(child.tag, child.attrib) 10 for i in child: 11 print(i.tag,i.text) 12 13 #只遍历year 节点 14 for node in root.iter('year'): 15 print(node.tag,node.text) 16 修改和删除xml文档内容
修改和删除xml文档内容
1 import xml.etree.ElementTree as ET 2 3 tree = ET.parse("xmltest.xml") 4 root = tree.getroot() 5 6 #修改 7 for node in root.iter('year'): 8 new_year = int(node.text) + 1 9 node.text = str(new_year) 10 node.set("updated","yes") 11 12 tree.write("xmltest.xml") 13 14 15 #删除node 16 for country in root.findall('country'): 17 rank = int(country.find('rank').text) 18 if rank > 50: 19 root.remove(country) 20 21 tree.write('output.xml')
自己创建xml文档
1 import xml.etree.ElementTree as ET 2 3 4 new_xml = ET.Element("namelist") 5 name = ET.SubElement(new_xml,"name",attrib={"enrolled":"yes"}) 6 age = ET.SubElement(name,"age",attrib={"checked":"no"}) 7 sex = ET.SubElement(name,"sex") 8 sex.text = '33' 9 name2 = ET.SubElement(new_xml,"name",attrib={"enrolled":"no"}) 10 age = ET.SubElement(name2,"age") 11 age.text = '19' 12 13 et = ET.ElementTree(new_xml) #生成文档对象 14 et.write("test.xml", encoding="utf-8",xml_declaration=True) 15 16 ET.dump(new_xml) #打印生成的格式
PyYAML模块
Python也可以很容易的处理ymal文档格式，只不过需要安装一个模块，参考文档：http://pyyaml.org/wiki/PyYAMLDocumentation
ConfigParser模块
用于生成和修改常见配置文档，当前模块的名称在 python 3.x 版本中变更为 configparser。
来看一个好多软件的常见文档格式如下：
[section]
option
[DEFAULT]
ServerAliveInterval = 45
Compression = yes
CompressionLevel = 9
ForwardX11 = yes
[bitbucket.org]
User = hg
[topsecret.server.com]
Port = 50022
ForwardX11 = no
如果想用python生成一个这样的文档怎么做呢？
import configparser config = configparser.ConfigParser() config["DEFAULT"] = {'ServerAliveInterval': '45', 'Compression': 'yes', 'CompressionLevel': '9'} config['bitbucket.org'] = {} config['bitbucket.org']['User'] = 'hg' config['topsecret.server.com'] = {} topsecret = config['topsecret.server.com'] topsecret['Host Port'] = '50022' # mutates the parser topsecret['ForwardX11'] = 'no' # same here config['DEFAULT']['ForwardX11'] = 'yes' with open('example.ini', 'w') as configfile: config.write(configfile)
写完了还可以再读出来
>>> import configparser >>> config = configparser.ConfigParser() >>> config.sections() [] >>> config.read('example.ini') ['example.ini'] >>> config.sections() ['bitbucket.org', 'topsecret.server.com'] >>> 'bitbucket.org' in config True >>> 'bytebong.com' in config False >>> config['bitbucket.org']['User'] 'hg' >>> config['DEFAULT']['Compression'] 'yes' >>> topsecret = config['topsecret.server.com'] >>> topsecret['ForwardX11'] 'no' >>> topsecret['Port'] '50022' >>> for key in config['bitbucket.org']: print(key) ... user compressionlevel serveraliveinterval compression forwardx11 >>> config['bitbucket.org']['ForwardX11'] 'yes'
configparser增删改查语法
import configparser config = configparser.ConfigParser() config.read('example.ini') # ########## 读 ########## # secs = config.sections() # print secs # options = config.options('group2') # print options # item_list = config.items('group2') # print item_list # val = config.get('group1','key') # val = config.getint('group1','key') # ########## 改写 ########## # sec = config.remove_section('group1') # config.write(open('example.ini', "w")) # sec = config.has_section('wupeiqi') # sec = config.add_section('wupeiqi') # config.write(open('example.ini', "w")) # config.set('group2','k1',11111) # config.write(open('example.ini', "w")) # config.remove_option('group2','age')
hashlib模块
hashlib是python的标准库，模块文件放在`find / -name hashlib.py`
python模块路径 `sys.path`，该路径下的模块文件可以直接导入模块
加密方式:
hashlib.md5(常用) hashlib.sha1 hashlib.sha256 hashlib.sha512
hashlib.new hashlib.sha224 hashlib.sha384
help(hashlib.md5)
Help on HASH object: class HASH(__builtin__.object) | A hash represents the object used to calculate a checksum of a | string of information. | | Methods: | | update() -- updates the current digest with an additional string | digest() -- return the current digest value | hexdigest() -- return the current digest as a string of hexadecimal digits | copy() -- return a copy of the current hash object | | Attributes: | | name -- the hash algorithm being used by this object | digest_size -- number of bytes in this hashes output | | Methods defined here: | | __repr__(...) | x.__repr__() <==> repr(x) | | copy(...) | Return a copy of the hash object. | | digest(...) | Return the digest value as a string of binary data. | | hexdigest(...) | Return the digest value as a string of hexadecimal digits. | | update(...) | Update this hash object's state with the provided string. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | block_size | | digest_size | | digestsize | | name | algorithm name.
md5加密的两种方法
第一种语法： In [2]: md5 = hashlib.md5() In [3]: md5.update('a') #追加一个字符串 In [4]: md5.hexdigest() Out[4]: '0cc175b9c0f1b6a831c399e269772661' [root@iZ288bqr3pkZ ~]# echo -n a | md5sum 0cc175b9c0f1b6a831c399e269772661 - In [10]: md5.update('b') In [11]: md5.hexdigest() Out[11]: '187ef4436122d1cc2f40dc2b92f0eba0' [root@iZ288bqr3pkZ ~]# echo -n ab | md5sum 187ef4436122d1cc2f40dc2b92f0eba0 - 第二种语法： In [9]: hashlib.md5('hello').hexdigest() Out[9]: '5d41402abc4b2a76b9719d911017c592' [root@iZ288bqr3pkZ ~]# echo -n hello | md5sum 5d41402abc4b2a76b9719d911017c592 -
加密文件内容
#!/usr/bin/env python import hashlib import sys import os def md5sum(f): m = hashlib.md5() with open(f) as fd: while True: data = fd.read(4096) #读4096字节 if data: #如果不为空 m.update(data) else: #如果不为空，到了行尾 break return m.hexdigest() if __name__ == '__main__': try: if os.path.exists(sys.argv[1]) == True: print md5sum(sys.argv[1]) else: print "%s file not empty" %sys.argv[1] except IndexError: print "%s follow a argument" %sys.argv[0]
[root@iZ288bqr3pkZ ~]# python test.py test.py
cfe496ca62d7ee20c3f6fbea33d68097
[root@iZ288bqr3pkZ ~]# python test.py
test.py follow a argument
walk模块
给目录下的文件加密
#!/usr/bin/env python #coding:utf8 import hashlib import sys import os def md5sum(f): m = hashlib.md5() with open(f) as fd: while True: data = fd.read(4096) #读4096字节 if data: #如果不为空 m.update(data) else: #如果不为空，到了行尾 break return m.hexdigest() a = os.walk(sys.argv[1]) for p,d,f in a: for i in f: fn = os.path.join(p,i) md5 = md5sum(fn) s += md5+' '+fn+'\n' print s,
[root@iZ288bqr3pkZ mulu]# tree . . ├── mulu1 │ ├── 1.txt │ └── 2.txt ├── mulu2 │ └── 3.txt └── test.py 2 directories, 4 files [root@iZ288bqr3pkZ mulu]# python test.py . 33751dd33c04ee5c9714918280a8f0f3 ./test.py d41d8cd98f00b204e9800998ecf8427e ./mulu2/3.txt d41d8cd98f00b204e9800998ecf8427e ./mulu1/2.txt d41d8cd98f00b204e9800998ecf8427e ./mulu1/1.txt [root@iZ288bqr3pkZ mulu]# find . -type f -exec md5sum {} \; 33751dd33c04ee5c9714918280a8f0f3 ./test.py d41d8cd98f00b204e9800998ecf8427e ./mulu2/3.txt d41d8cd98f00b204e9800998ecf8427e ./mulu1/2.txt d41d8cd98f00b204e9800998ecf8427e ./mulu1/1.txt
字典排序
import operator x = {1:2, 3:4, 4:3, 2:1, 0:0} #按key排序 sorted_x = sorted(x.iteritems(), key=operator.itemgetter(0)) sorted_x [(0, 0), (1, 2), (2, 1), (3, 4), (4, 3)] #按value排序 sorted_x = sorted(x.iteritems(), key=operator.itemgetter(1)) sorted_x [(0, 0), (2, 1), (1, 2), (4, 3), (3, 4)] #逆序排序 sorted_y = sorted(x.iteritems(), key=operator.itemgetter(1), reverse=True) sorted_y [(3, 4), (4, 3), (1, 2), (2, 1), (0, 0)]
找出占用空间大的文件
#!/usr/bin/env python import os import sys import operator def gen_dic(topdir): dic = {} a = os.walk(topdir) for p, d, f in a: for i in f: fn = os.path.join(p, i) f_size = os.path.getsize(fn) dic[fn] = f_size return dic if __name__ == '__main__': dic = gen_dic(sys.argv[1]) sorted_dic = sorted(dic.iteritems(), key=operator.itemgetter(1), reverse=True) for k, v in sorted_dic[:10]: print k, '-->', v
subprocess模块
可以执行shell命令的相关模块和函数有：
- os.system
- os.spawn*
- os.popen* --废弃
- popen2.* --废弃
- commands.* --废弃，3.x中被移除
import commands
result = commands.getoutput('cmd')
result = commands.getstatus('cmd')
result = commands.getstatusoutput('cmd')
以上执行shell命令的相关的模块和函数的功能均在 subprocess 模块中实现，并提供了更丰富的功能。
subprocess开启一个子进程去执行命令
shell=True 表示直接用shell解释命令shell=False（默认）表示用python解释
subprocess.call
#!/usr/bin/env python import subprocess try: subprocess.call('ls l ', shell=True) except Exception,e: print e print 'hello world' [root@iZ288bqr3pkZ ~]# python test.py ls: cannot access l: No such file or directory hello world
subprocess.check_call：可捕获异常
#!/usr/bin/env python import subprocess try: subprocess.check_call('ls l ', shell=True) 2 except Exception,e: print e print 'hello world' [root@iZ288bqr3pkZ ~]# python test.py ls: cannot access l: No such file or directory Command 'ls l ' returned non-zero exit status 2 hello world
subprocess.Popen
可用参数：
- args：shell命令，可以是字符串或者序列类型（如：list，元组）
- bufsize：指定缓冲。0 无缓冲,1 行缓冲,其他 缓冲区大小,负值 系统缓冲
- stdin, stdout, stderr：分别表示程序的标准输入、输出、错误句柄
- preexec_fn：只在Unix平台下有效，用于指定一个可执行对象（callable object），它将在子进程运行之前被调用
- close_sfs：在windows平台下，如果close_fds被设置为True，则新创建的子进程将不会继承父进程的输入、输出、错误管道。
所以不能将close_fds设置为True同时重定向子进程的标准输入、输出与错误(stdin, stdout, stderr)。 - shell：同上
- cwd：用于设置子进程的当前目录
- env：用于指定子进程的环境变量。如果env = None，子进程的环境变量将从父进程中继承。
- universal_newlines：不同系统的换行符不同，True -> 同意使用 \n
- startupinfo与createionflags只在windows下有效
将被传递给底层的CreateProcess()函数，用于设置子进程的一些属性，如：主窗口的外观，进程的优先级等等
终端输入的命令分为两种：
- 输入即可得到输出，如：ifconfig
- 输入进行某环境，依赖再输入，如：python
需要交互的命令示例
popen默认不会等待子进程执行完，用于返回命令执行结果
test.py
#!/usr/bin/env python import subprocess try: subprocess.call('ls', shell=True) except Exception,e: print e print 'hello world'
test1.py
#!/usr/bin/env python from subprocess import Popen, PIPE p = Popen('./test.py',shell=True) p.wait() print "main process" [root@iZ288bqr3pkZ ~]# python test1.py 4_top10.py test1.py test.py hello world main process
PIPE ：PIPE管道可用于父进程和子进程交互
In [6]: from subprocess import popen,pipe In [6]: from subprocess import Popen,PIPE In [7]: p1 = Popen(['ls'], stdout=PIPE) In [8]: p2 = Popen(['grep', 'py'], stdin=p1.stdout, stdout=PIPE) In [9]: result = p2.stdout In [10]: for i in result:print i, 4_top10.py test1.py test.py
import subprocess obj = subprocess.Popen(["python"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) obj.stdin.write('print 1 \n ') obj.stdin.write('print 2 \n ') obj.stdin.write('print 3 \n ') obj.stdin.write('print 4 \n ') out_error_list = obj.communicate(timeout=10) print out_error_list
判断一个字符串是数字
#!/usr/bin/env python import os import sys import string def isNum(s): for i in s: if i in string.digits: continue else: return False return True for pid in os.listdir('/proc'): if isNum(pid): print pid
统计apache进程占用的物理内存
#!/usr/bin/env python import os from subprocess import Popen, PIPE def getPid(): p = Popen(['pidof', 'httpd'], stdout=PIPE, stderr=PIPE) pids = p.stdout.read().split() return pids def parsePidFile(pids): sum = 0 for i in pids: fn = os.path.join('/proc/',i,'status') with open(fn) as fd: for line in fd: if line.startswith('VmRSS'): http_mem = int(line.split()[1]) sum += http_mem break return sum def total_mem(f): with open(f) as fd: for line in fd: if line.startswith('MemTotal'): total_mem = int(line.split()[1]) return total_mem if __name__ == '__main__': pids = getPid() http_mem = parsePidFile(pids) total = total_mem('/proc/meminfo') print "Apache memory is: %s KB" % http_mem print "Percent: %.2f%%" % (http_mem/float(total)*100) print "Percent: %.2f" % (http_mem/float(total)*100)+'%'
Dmidecode系统信息
#!/usr/bin/env python #coding:utf8 from subprocess import Popen, PIPE p = Popen(['dmidecode'], stdout=PIPE) data = p.stdout lines = [] dmi = {} a = True while a: line = data.readline() if line.startswith('System Information'): while True: line = data.readline() if line == '\n': a = False break else: lines.append(line) dmi_dic = dict([i.strip().split(':') for i in lines]) dmi['Manufacturer'] = dmi_dic['Manufacturer'].strip() dmi['Product'] = dmi_dic['Product Name'].strip() dmi['Serial'] = dmi_dic['Serial Number'].strip() print dmi
收集IP信息
#!/usr/bin/env python from subprocess import Popen, PIPE def getIfconfig(): p = Popen(['ifconfig'], stdout=PIPE) data = p.stdout.read().split('\n\n') return [i for i in data if i and not i.startswith('lo')] def parseIfconfig(data): dic = {} for lines in data: line_list = lines.split('\n') devname = line_list[0].split()[0] macaddr = line_list[0].split()[-1] ipaddr = line_list[1].split()[1].split(':')[1] dic[devname] = [ipaddr, macaddr] return dic if __name__ == '__main__': data = getIfconfig() print parseIfconfig(data)
#!/usr/bin/env python #coding:utf8 #通用windows unix from subprocess import Popen, PIPE def getIP(): p = Popen('ifconfig',shell=True, stdout=PIPE, stderr=PIPE) #p.communicate()方法相当于p.stdin.write()、p.stdin.close()与p.stdout.read()这3个方法 stdout, stderr = p.communicate() data = [i for i in stdout.split('\n') if i] return data def genIP(data): new_line = '' lines = [] for line in data: if line[0].strip(): #如果是顶行，把之前累加的行成为列表的一个元素 lines.append(new_line) #从新的顶行开始赋值 new_line = line + '\n' else: #如果是缩进行，累加行成一行 new_line += line + '\n' lines.append(new_line) #由于累加完后，遇不到顶行，最后一部分要手动追加 return [i for i in lines if i and not i.startswith('lo')] def parseIfconfig(data): dic = {} for lines in data: line_list = lines.split('\n') devname = line_list[0].split()[0] macaddr = line_list[0].split()[-1] ipaddr = line_list[1].split()[1].split(':')[1] dic[devname] = [ipaddr, macaddr] return dic if __name__ == '__main__': data = getIP() data_list = genIP(data) print parseIfconfig(data_list)
收集主机信息
1. 主机名：hostname 2. IP地址：ip 3. 操作系统版本：osver 4. 服务器厂商：vendor 5. 服务器型号：product 6. 服务器序列号：sn 7. cpu型号：cpu_model 8. cpu核数：cpu_num 9. 内存大小：memory
#!/usr/bin/env python import urllib, urllib2 from subprocess import Popen, PIPE import pickle def getIfconfig(): p = Popen(['ifconfig'], stdout=PIPE) data = p.stdout.read() return data def getDmi(): p = Popen(['dmidecode'], stdout=PIPE) data = p.stdout.read() return data #把数据放进列表 def parseData(data): parsed_data = [] new_line = '' data = [i for i in data.split('\n') if i] for line in data: if line[0].strip(): parsed_data.append(new_line) new_line = line+'\n' else: new_line += line+'\n' parsed_data.append(new_line) return [i for i in parsed_data if i] def parseIfconfig(parsed_data): dic = {} parsed_data = [i for i in parsed_data if not i.startswith('lo')] for lines in parsed_data: line_list = lines.split('\n') devname = line_list[0].split()[0] macaddr = line_list[0].split()[-1] ipaddr = line_list[1].split()[1].split(':')[1] break dic['ip'] = ipaddr return dic def parseDmi(parsed_data): dic = {} parsed_data = [i for i in parsed_data if i.startswith('System Information')] parsed_data = [i for i in parsed_data[0].split('\n')[1:] if i] dmi_dic = dict([i.strip().split(':') for i in parsed_data]) dic['vendor'] = dmi_dic['Manufacturer'].strip() dic['product'] = dmi_dic['Product Name'].strip() dic['sn'] = dmi_dic['Serial Number'].strip()[:15] return dic def getHostname(f): with open(f) as fd: for line in fd: if line.startswith('HOSTNAME'): hostname = line.split('=')[1].strip() break return {'hostname':hostname} def getOSver(f): with open(f) as fd: for line in fd: osver = line.strip() break return {'osver':osver} def getCpu(f): num = 0 with open(f) as fd: for line in fd: if line.startswith('processor'): num += 1 if line.startswith('model name'): cpu_model = line.split(':')[1].split() cpu_model = cpu_model[0]+' '+cpu_model[-1] return {'cpu_num':num, 'cpu_model':cpu_model} def getMemory(f): with open(f) as fd: for line in fd: if line.startswith('MemTotal'): mem = int(line.split()[1].strip()) break mem = "%s" % int(mem/1024.0)+'M' return {'memory':mem} if __name__ == '__main__': dic = {} data_ip = getIfconfig() parsed_data_ip = parseData(data_ip) ip = parseIfconfig(parsed_data_ip) data_dmi = getDmi() parsed_data_dmi = parseData(data_dmi) dmi = parseDmi(parsed_data_dmi) hostname = getHostname('/etc/sysconfig/network') osver = getOSver('/etc/issue') cpu = getCpu('/proc/cpuinfo') mem = getMemory('/proc/meminfo') dic.update(ip) dic.update(dmi) dic.update(hostname) dic.update(osver) dic.update(cpu) dic.update(mem) print dic #d = urllib.urlencode(dic) d = pickle.dumps(dic) req = urllib2.urlopen('http://192.168.1.5:8000/hostinfo/collect/',d) print req.read()
logging模块
很多程序都有记录日志的需求，并且日志中包含的信息即有正常的程序访问日志，还可能有错误、警告等信息输出，python的logging模块提供了标准的日志接口，你可以通过它存储各种格式的日志，logging的日志可以分为 debug(), info(), warning(), error() and critical() 5个级别，下面我们看一下怎么用。
最简单用法
import logging
logging.warning("user [hy] attempted wrong password more than 3 times")
logging.critical("server is down")
#输出
WARNING:root:user [alex] attempted wrong password more than 3 times
CRITICAL:root:server is down
日志级别：DEBUG INFO WARNING ERROR CRITICAL
把日志写入到文件
import logging
logging.basicConfig(filename='example.log', level=logging.INFO,
format='[%(asctime)s %(filename)s]: %(levelname)s %(message)s ', datefmt='%m/%d/%Y %H:%M:%S:')
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
同时把log打印在屏幕和文件日志
全局logger级别和StreamHandler、FileHandler级别，相比哪个级别高，用哪个级别
import logging
#create logger
logger = logging.getLogger('TEST-LOG')
logger.setLevel(logging.DEBUG)
# create console handler and set level to debug
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# create file handler and set level to warning
fh = logging.FileHandler("access.log")
fh.setLevel(logging.WARNING)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# add formatter to ch and fh
ch.setFormatter(formatter)
fh.setFormatter(formatter)
# add ch and fh to logger
logger.addHandler(ch)
logger.addHandler(fh)
# 'application' code
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')re模块
支持匹配字符串r和字节b
http://www.cnblogs.com/hyit/articles/4727760.html
re.DEBUG re.M re.U re.error re.search re.sys
re.DOTALL re.MULTILINE re.UNICODE re.escape re.split re.template
re.I re.S re.VERBOSE re.findall re.sre_compile
re.IGNORECASE re.Scanner re.X re.finditer re.sre_parse
re.L re.T re.compile re.match re.sub
re.LOCALE re.TEMPLATE re.copy_reg re.purge re.subn
re.M //识别\n，将一个字符串按\n分成多个
In [1]: import re In [2]: data = 'foo1\nfoo2\n' In [3]: re.findall(r'foo.$',data) Out[3]: ['foo2'] In [4]: re.findall(r'foo.$',data,re.M) Out[4]: ['foo1', 'foo2']
收集网卡信息
#!/usr/bin/env python import re from subprocess import Popen, PIPE def getIfconfig(): p = Popen(['ifconfig'], stdout=PIPE) data = p.stdout.read().split('\n\n') return [i for i in data if i and not i.startswith('lo')] def parseIfconfig(data): re_devname = re.compile(r'(br|eth|em|virbr|lo|bond)[\d:]+',re.M) re_mac = re.compile(r'HWaddr ([0-9A-F:]{17})', re.M) re_ip = re.compile(r'inet addr:([\d\.]{7,15})', re.M) devname = re_devname.search(data) if devname: devname = devname.group() else: devname = '' mac = re_mac.search(data) if mac: mac = mac.group(1) else: mac = '' ip = re_ip.search(data) if ip: ip = ip.group(1) else: ip = '' return {devname: [ip, mac]} if __name__ == '__main__': dic = {} data = getIfconfig() for i in data: dic.update(parseIfconfig(i)) print dic
paramiko模块
作用：不需要安装ftp等服务，就可以：执行命令，上传、下载文件
import paramiko # 创建SSH对象 ssh = paramiko.SSHClient() # 允许连接不在know_hosts文件中的主机 ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) # 连接服务器 ssh.connect(hostname='ip', port=22, username='root', password='password') # 执行命令 stdin, stdout, stderr = ssh.exec_command('df') # 获取命令结果 result = stdout.read() print result # 关闭连接 ssh.close() 基于用户名密码连接：
import paramiko transport = paramiko.Transport(('ip', 22)) transport.connect(username='root', password='password') ssh = paramiko.SSHClient() ssh._transport = transport stdin, stdout, stderr = ssh.exec_command('df') print stdout.read() transport.close()
import paramiko #先建立好密钥：http://www.cnblogs.com/hyit/articles/5090218.html private_key = paramiko.RSAKey.from_private_key_file('/root/.ssh/id_rsa') # 创建SSH对象 ssh = paramiko.SSHClient() # 允许连接不在know_hosts文件中的主机 ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) # 连接服务器 #如果key有密码,加password='password' ssh.connect(hostname='ip', port=22, username='root',pkey=private_key) # 执行命令 stdin, stdout, stderr = ssh.exec_command('df') # 获取命令结果 result = stdout.read() print result # 关闭连接 ssh.close()
import paramiko private_key = paramiko.RSAKey.from_private_key_file('/root/.ssh/id_rsa') transport = paramiko.Transport(('ip', 22)) transport.connect(username='root', pkey=private_key) ssh = paramiko.SSHClient() ssh._transport = transport stdin, stdout, stderr = ssh.exec_command('df') transport.close()
import paramiko transport = paramiko.Transport(('ip', 22)) transport.connect(username='root', password='password') sftp = paramiko.SFTPClient.from_transport(transport) # 将location.py 上传至服务器 /tmp/test.py sftp.put('test.py', '/tmp/test.py') # 将remove_path 下载到本地 local_path sftp.get('/tmp/zabbix_server.log', '/tmp/zabbix_server.log')
import paramiko private_key = paramiko.RSAKey.from_private_key_file('/root/.ssh/id_rsa') transport = paramiko.Transport(('139.129.119.104', 22)) transport.connect(username='root', pkey=private_key ) sftp = paramiko.SFTPClient.from_transport(transport) # 将location.py 上传至服务器 /tmp/test.py sftp.put('test.py', '/tmp/test.py') # 将remove_path 下载到本地 local_path sftp.get('/tmp/zabbix_server.log', '/tmp/zabbix_server.log') transport.close()
websocket模块
官方文档 https://github.com/websocket-client/websocket-client
WebSocket协议是基于TCP的一种新的协议。WebSocket最初在HTML5规范中被引用为TCP连接，作为基于TCP的套接字API的占位符。它实现了浏览器与服务器全双工(full-duplex)通信。其本质是保持TCP连接，在浏览器和服务端通过Socket进行通信。为了补充http协议而生（http单工通信和非长连接的特性无法满足更多需求）
概述
使用websocket有三种形式，create_connection、WebSocket、WebSocketApp(WebSocketApp是更高级的封装形式)，
create_connection和WebSocket是基础版更灵活，语法基本一样
WebSocketApp添加了事件处理函数，更高级的形式，编码更方便
其实，使用哪个这个完全取决于你的偏好源码剖析
class WebSocket(object): """ Low level WebSocket interface. This class is based on The WebSocket protocol draft-hixie-thewebsocketprotocol-76 http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-76 We can connect to the websocket server and send/receive data. The following example is an echo client. >>> import websocket >>> ws = websocket.WebSocket() >>> ws.connect("ws://echo.websocket.org") >>> ws.send("Hello, Server") >>> ws.recv() 'Hello, Server' >>> ws.close() get_mask_key: a callable to produce new mask keys, see the set_mask_key function's docstring for more details sockopt: values for socket.setsockopt. sockopt must be tuple and each element is argument of sock.setsockopt. sslopt: dict object for ssl socket option. fire_cont_frame: fire recv event for each cont frame. default is False enable_multithread: if set to True, lock send method. skip_utf8_validation: skip utf8 validation. """ def __init__(self, get_mask_key=None, sockopt=None, sslopt=None, fire_cont_frame=False, enable_multithread=False, skip_utf8_validation=False, **_): """ Initialize WebSocket object. """ self.sock_opt = sock_opt(sockopt, sslopt) self.handshake_response = None self.sock = None self.connected = False self.get_mask_key = get_mask_key # These buffer over the build-up of a single frame. self.frame_buffer = frame_buffer(self._recv, skip_utf8_validation) self.cont_frame = continuous_frame( fire_cont_frame, skip_utf8_validation) if enable_multithread: self.lock = threading.Lock() self.readlock = threading.Lock() else: self.lock = NoLock() self.readlock = NoLock() def __iter__(self): """ Allow iteration over websocket, implying sequential `recv` executions. """ while True: yield self.recv() def __next__(self): return self.recv() def next(self): return self.__next__() def fileno(self): return self.sock.fileno() def set_mask_key(self, func): """ set function to create musk key. You can customize mask key generator. Mainly, this is for testing purpose. func: callable object. the func takes 1 argument as integer. The argument means length of mask key. This func must return string(byte array), which length is argument specified. """ self.get_mask_key = func def gettimeout(self): """ Get the websocket timeout(second). """ return self.sock_opt.timeout def settimeout(self, timeout): """ Set the timeout to the websocket. timeout: timeout time(second). """ self.sock_opt.timeout = timeout if self.sock: self.sock.settimeout(timeout) timeout = property(gettimeout, settimeout) def getsubprotocol(self): """ get subprotocol """ if self.handshake_response: return self.handshake_response.subprotocol else: return None subprotocol = property(getsubprotocol) def getstatus(self): """ get handshake status """ if self.handshake_response: return self.handshake_response.status else: return None status = property(getstatus) def getheaders(self): """ get handshake response header """ if self.handshake_response: return self.handshake_response.headers else: return None def is_ssl(self): return isinstance(self.sock, ssl.SSLSocket) headers = property(getheaders) def connect(self, url, **options): """ Connect to url. url is websocket url scheme. ie. ws://host:port/resource You can customize using 'options'. If you set "header" list object, you can set your own custom header. >>> ws = WebSocket() >>> ws.connect("ws://echo.websocket.org/", ... header=["User-Agent: MyProgram", ... "x-custom: header"]) timeout: socket timeout time. This value is integer. if you set None for this value, it means "use default_timeout value" options: "header" -> custom http header list or dict. "cookie" -> cookie value. "origin" -> custom origin url. "suppress_origin" -> suppress outputting origin header. "host" -> custom host header string. "http_proxy_host" - http proxy host name. "http_proxy_port" - http proxy port. If not set, set to 80. "http_no_proxy" - host names, which doesn't use proxy. "http_proxy_auth" - http proxy auth information. tuple of username and password. default is None "redirect_limit" -> number of redirects to follow. "subprotocols" - array of available sub protocols. default is None. "socket" - pre-initialized stream socket. """ # FIXME: "subprotocols" are getting lost, not passed down # FIXME: "header", "cookie", "origin" and "host" too self.sock_opt.timeout = options.get('timeout', self.sock_opt.timeout) self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options), options.pop('socket', None)) try: self.handshake_response = handshake(self.sock, *addrs, **options) for attempt in range(options.pop('redirect_limit', 3)): if self.handshake_response.status in SUPPORTED_REDIRECT_STATUSES: url = self.handshake_response.headers['location'] self.sock.close() self.sock, addrs = connect(url, self.sock_opt, proxy_info(**options), options.pop('socket', None)) self.handshake_response = handshake(self.sock, *addrs, **options) self.connected = True except: if self.sock: self.sock.close() self.sock = None raise def send(self, payload, opcode=ABNF.OPCODE_TEXT): """ Send the data as string. payload: Payload must be utf-8 string or unicode, if the opcode is OPCODE_TEXT. Otherwise, it must be string(byte array) opcode: operation code to send. Please see OPCODE_XXX. """ frame = ABNF.create_frame(payload, opcode) return self.send_frame(frame) def send_frame(self, frame): """ Send the data frame. frame: frame data created by ABNF.create_frame >>> ws = create_connection("ws://echo.websocket.org/") >>> frame = ABNF.create_frame("Hello", ABNF.OPCODE_TEXT) >>> ws.send_frame(frame) >>> cont_frame = ABNF.create_frame("My name is ", ABNF.OPCODE_CONT, 0) >>> ws.send_frame(frame) >>> cont_frame = ABNF.create_frame("Foo Bar", ABNF.OPCODE_CONT, 1) >>> ws.send_frame(frame) """ if self.get_mask_key: frame.get_mask_key = self.get_mask_key data = frame.format() length = len(data) trace("send: " + repr(data)) with self.lock: while data: l = self._send(data) data = data[l:] return length def send_binary(self, payload): return self.send(payload, ABNF.OPCODE_BINARY) def ping(self, payload=""): """ send ping data. payload: data payload to send server. """ if isinstance(payload, six.text_type): payload = payload.encode("utf-8") self.send(payload, ABNF.OPCODE_PING) def pong(self, payload): """ send pong data. payload: data payload to send server. """ if isinstance(payload, six.text_type): payload = payload.encode("utf-8") self.send(payload, ABNF.OPCODE_PONG) def recv(self): """ Receive string data(byte array) from the server. return value: string(byte array) value. """ with self.readlock: opcode, data = self.recv_data() if six.PY3 and opcode == ABNF.OPCODE_TEXT: return data.decode("utf-8") elif opcode == ABNF.OPCODE_TEXT or opcode == ABNF.OPCODE_BINARY: return data else: return '' def recv_data(self, control_frame=False): """ Receive data with operation code. control_frame: a boolean flag indicating whether to return control frame data, defaults to False return value: tuple of operation code and string(byte array) value. """ opcode, frame = self.recv_data_frame(control_frame) return opcode, frame.data def recv_data_frame(self, control_frame=False): """ Receive data with operation code. control_frame: a boolean flag indicating whether to return control frame data, defaults to False return value: tuple of operation code and string(byte array) value. """ while True: frame = self.recv_frame() if not frame: # handle error: # 'NoneType' object has no attribute 'opcode' raise WebSocketProtocolException( "Not a valid frame %s" % frame) elif frame.opcode in (ABNF.OPCODE_TEXT, ABNF.OPCODE_BINARY, ABNF.OPCODE_CONT): self.cont_frame.validate(frame) self.cont_frame.add(frame) if self.cont_frame.is_fire(frame): return self.cont_frame.extract(frame) elif frame.opcode == ABNF.OPCODE_CLOSE: self.send_close() return frame.opcode, frame elif frame.opcode == ABNF.OPCODE_PING: if len(frame.data) < 126: self.pong(frame.data) else: raise WebSocketProtocolException( "Ping message is too long") if control_frame: return frame.opcode, frame elif frame.opcode == ABNF.OPCODE_PONG: if control_frame: return frame.opcode, frame def recv_frame(self): """ receive data as frame from server. return value: ABNF frame object. """ return self.frame_buffer.recv_frame() def send_close(self, status=STATUS_NORMAL, reason=six.b("")): """ send close data to the server. status: status code to send. see STATUS_XXX. reason: the reason to close. This must be string or bytes. """ if status < 0 or status >= ABNF.LENGTH_16: raise ValueError("code is invalid range") self.connected = False self.send(struct.pack('!H', status) + reason, ABNF.OPCODE_CLOSE) def close(self, status=STATUS_NORMAL, reason=six.b(""), timeout=3): """ Close Websocket object status: status code to send. see STATUS_XXX. reason: the reason to close. This must be string. timeout: timeout until receive a close frame. If None, it will wait forever until receive a close frame. """ if self.connected: if status < 0 or status >= ABNF.LENGTH_16: raise ValueError("code is invalid range") try: self.connected = False self.send(struct.pack('!H', status) + reason, ABNF.OPCODE_CLOSE) sock_timeout = self.sock.gettimeout() self.sock.settimeout(timeout) start_time = time.time() while timeout is None or time.time() - start_time < timeout: try: frame = self.recv_frame() if frame.opcode != ABNF.OPCODE_CLOSE: continue if isEnabledForError(): recv_status = struct.unpack("!H", frame.data[0:2])[0] if recv_status != STATUS_NORMAL: error("close status: " + repr(recv_status)) break except: break self.sock.settimeout(sock_timeout) self.sock.shutdown(socket.SHUT_RDWR) except: pass self.shutdown() def abort(self): """ Low-level asynchronous abort, wakes up other threads that are waiting in recv_* """ if self.connected: self.sock.shutdown(socket.SHUT_RDWR) def shutdown(self): """close socket, immediately.""" if self.sock: self.sock.close() self.sock = None self.connected = False def _send(self, data): return send(self.sock, data) def _recv(self, bufsize): try: return recv(self.sock, bufsize) except WebSocketConnectionClosedException: if self.sock: self.sock.close() self.sock = None self.connected = False raise
class WebSocketApp(object): """ Higher level of APIs are provided. The interface is like JavaScript WebSocket object. """ def __init__(self, url, header=None, on_open=None, on_message=None, on_error=None, on_close=None, on_ping=None, on_pong=None, on_cont_message=None, keep_running=True, get_mask_key=None, cookie=None, subprotocols=None, on_data=None): """ url: websocket url. header: custom header for websocket handshake. on_open: callable object which is called at opening websocket. this function has one argument. The argument is this class object. on_message: callable object which is called when received data. on_message has 2 arguments. The 1st argument is this class object. The 2nd argument is utf-8 string which we get from the server. on_error: callable object which is called when we get error. on_error has 2 arguments. The 1st argument is this class object. The 2nd argument is exception object. on_close: callable object which is called when closed the connection. this function has one argument. The argument is this class object. on_cont_message: callback object which is called when receive continued frame data. on_cont_message has 3 arguments. The 1st argument is this class object. The 2nd argument is utf-8 string which we get from the server. The 3rd argument is continue flag. if 0, the data continue to next frame data on_data: callback object which is called when a message received. This is called before on_message or on_cont_message, and then on_message or on_cont_message is called. on_data has 4 argument. The 1st argument is this class object. The 2nd argument is utf-8 string which we get from the server. The 3rd argument is data type. ABNF.OPCODE_TEXT or ABNF.OPCODE_BINARY will be came. The 4th argument is continue flag. if 0, the data continue keep_running: this parameter is obsolete and ignored. get_mask_key: a callable to produce new mask keys, see the WebSocket.set_mask_key's docstring for more information subprotocols: array of available sub protocols. default is None. """ self.url = url self.header = header if header is not None else [] self.cookie = cookie self.on_open = on_open self.on_message = on_message self.on_data = on_data self.on_error = on_error self.on_close = on_close self.on_ping = on_ping self.on_pong = on_pong self.on_cont_message = on_cont_message self.keep_running = False self.get_mask_key = get_mask_key self.sock = None self.last_ping_tm = 0 self.last_pong_tm = 0 self.subprotocols = subprotocols def send(self, data, opcode=ABNF.OPCODE_TEXT): """ send message. data: message to send. If you set opcode to OPCODE_TEXT, data must be utf-8 string or unicode. opcode: operation code of data. default is OPCODE_TEXT. """ if not self.sock or self.sock.send(data, opcode) == 0: raise WebSocketConnectionClosedException( "Connection is already closed.") def close(self, **kwargs): """ close websocket connection. """ self.keep_running = False if self.sock: self.sock.close(**kwargs) self.sock = None def _send_ping(self, interval, event): while not event.wait(interval): self.last_ping_tm = time.time() if self.sock: try: self.sock.ping() except Exception as ex: _logging.warning("send_ping routine terminated: {}".format(ex)) break def run_forever(self, sockopt=None, sslopt=None, ping_interval=0, ping_timeout=None, http_proxy_host=None, http_proxy_port=None, http_no_proxy=None, http_proxy_auth=None, skip_utf8_validation=False, host=None, origin=None, dispatcher=None, suppress_origin = False, proxy_type=None): """ run event loop for WebSocket framework. This loop is infinite loop and is alive during websocket is available. sockopt: values for socket.setsockopt. sockopt must be tuple and each element is argument of sock.setsockopt. sslopt: ssl socket optional dict. ping_interval: automatically send "ping" command every specified period(second) if set to 0, not send automatically. ping_timeout: timeout(second) if the pong message is not received. http_proxy_host: http proxy host name. http_proxy_port: http proxy port. If not set, set to 80. http_no_proxy: host names, which doesn't use proxy. skip_utf8_validation: skip utf8 validation. host: update host header. origin: update origin header. dispatcher: customize reading data from socket. suppress_origin: suppress outputting origin header. Returns ------- False if caught KeyboardInterrupt True if other exception was raised during a loop """ if ping_timeout is not None and ping_timeout <= 0: ping_timeout = None if ping_timeout and ping_interval and ping_interval <= ping_timeout: raise WebSocketException("Ensure ping_interval > ping_timeout") if not sockopt: sockopt = [] if not sslopt: sslopt = {} if self.sock: raise WebSocketException("socket is already opened") thread = None self.keep_running = True self.last_ping_tm = 0 self.last_pong_tm = 0 def teardown(close_frame=None): """ Tears down the connection. If close_frame is set, we will invoke the on_close handler with the statusCode and reason from there. """ if thread and thread.isAlive(): event.set() thread.join() self.keep_running = False if self.sock: self.sock.close() close_args = self._get_close_args( close_frame.data if close_frame else None) self._callback(self.on_close, *close_args) self.sock = None try: self.sock = WebSocket( self.get_mask_key, sockopt=sockopt, sslopt=sslopt, fire_cont_frame=self.on_cont_message is not None, skip_utf8_validation=skip_utf8_validation, enable_multithread=True if ping_interval else False) self.sock.settimeout(getdefaulttimeout()) self.sock.connect( self.url, header=self.header, cookie=self.cookie, http_proxy_host=http_proxy_host, http_proxy_port=http_proxy_port, http_no_proxy=http_no_proxy, http_proxy_auth=http_proxy_auth, subprotocols=self.subprotocols, host=host, origin=origin, suppress_origin=suppress_origin, proxy_type=proxy_type) if not dispatcher: dispatcher = self.create_dispatcher(ping_timeout) self._callback(self.on_open) if ping_interval: event = threading.Event() thread = threading.Thread( target=self._send_ping, args=(ping_interval, event)) thread.setDaemon(True) thread.start() def read(): if not self.keep_running: return teardown() op_code, frame = self.sock.recv_data_frame(True) if op_code == ABNF.OPCODE_CLOSE: return teardown(frame) elif op_code == ABNF.OPCODE_PING: self._callback(self.on_ping, frame.data) elif op_code == ABNF.OPCODE_PONG: self.last_pong_tm = time.time() self._callback(self.on_pong, frame.data) elif op_code == ABNF.OPCODE_CONT and self.on_cont_message: self._callback(self.on_data, frame.data, frame.opcode, frame.fin) self._callback(self.on_cont_message, frame.data, frame.fin) else: data = frame.data if six.PY3 and op_code == ABNF.OPCODE_TEXT: data = data.decode("utf-8") self._callback(self.on_data, data, frame.opcode, True) self._callback(self.on_message, data) return True def check(): if (ping_timeout): has_timeout_expired = time.time() - self.last_ping_tm > ping_timeout has_pong_not_arrived_after_last_ping = self.last_pong_tm - self.last_ping_tm < 0 has_pong_arrived_too_late = self.last_pong_tm - self.last_ping_tm > ping_timeout if (self.last_ping_tm and has_timeout_expired and (has_pong_not_arrived_after_last_ping or has_pong_arrived_too_late)): raise WebSocketTimeoutException("ping/pong timed out") return True dispatcher.read(self.sock.sock, read, check) except (Exception, KeyboardInterrupt, SystemExit) as e: self._callback(self.on_error, e) if isinstance(e, SystemExit): # propagate SystemExit further raise teardown() return not isinstance(e, KeyboardInterrupt) def create_dispatcher(self, ping_timeout): timeout = ping_timeout or 10 if self.sock.is_ssl(): return SSLDispacther(self, timeout) return Dispatcher(self, timeout) def _get_close_args(self, data): """ this functions extracts the code, reason from the close body if they exists, and if the self.on_close except three arguments """ # if the on_close callback is "old", just return empty list if sys.version_info < (3, 0): if not self.on_close or len(inspect.getargspec(self.on_close).args) != 3: return [] else: if not self.on_close or len(inspect.getfullargspec(self.on_close).args) != 3: return [] if data and len(data) >= 2: code = 256 * six.byte2int(data[0:1]) + six.byte2int(data[1:2]) reason = data[2:].decode('utf-8') return [code, reason] return [None, None] def _callback(self, callback, *args): if callback: try: if inspect.ismethod(callback): callback(*args) else: callback(self, *args) except Exception as e: _logging.error("error from callback {}: {}".format(callback, e)) if _logging.isEnabledForDebug(): _, _, tb = sys.exc_info() traceback.print_tb(tb)
def create_connection(url, timeout=None, class_=WebSocket, **options): """ connect to url and return websocket object. Connect to url and return the WebSocket object. Passing optional timeout parameter will set the timeout on the socket. If no timeout is supplied, the global default timeout setting returned by getdefauttimeout() is used. You can customize using 'options'. If you set "header" list object, you can set your own custom header. >>> conn = create_connection("ws://echo.websocket.org/", ... header=["User-Agent: MyProgram", ... "x-custom: header"]) timeout: socket timeout time. This value is integer. if you set None for this value, it means "use default_timeout value" class_: class to instantiate when creating the connection. It has to implement settimeout and connect. It's __init__ should be compatible with WebSocket.__init__, i.e. accept all of it's kwargs. options: "header" -> custom http header list or dict. "cookie" -> cookie value. "origin" -> custom origin url. "suppress_origin" -> suppress outputting origin header. "host" -> custom host header string. "http_proxy_host" - http proxy host name. "http_proxy_port" - http proxy port. If not set, set to 80. "http_no_proxy" - host names, which doesn't use proxy. "http_proxy_auth" - http proxy auth information. tuple of username and password. default is None "enable_multithread" -> enable lock for multithread. "redirect_limit" -> number of redirects to follow. "sockopt" -> socket options "sslopt" -> ssl option "subprotocols" - array of available sub protocols. default is None. "skip_utf8_validation" - skip utf8 validation. "socket" - pre-initialized stream socket. """ sockopt = options.pop("sockopt", []) sslopt = options.pop("sslopt", {}) fire_cont_frame = options.pop("fire_cont_frame", False) enable_multithread = options.pop("enable_multithread", False) skip_utf8_validation = options.pop("skip_utf8_validation", False) websock = class_(sockopt=sockopt, sslopt=sslopt, fire_cont_frame=fire_cont_frame, enable_multithread=enable_multithread, skip_utf8_validation=skip_utf8_validation, **options) websock.settimeout(timeout if timeout is not None else getdefaulttimeout()) websock.connect(url, **options) return websock
自定义websocket server
#!/usr/bin/env python # -*- coding:utf-8 -*- import socket import base64 import hashlib def get_headers(data): """ 将请求头格式化成字典 :param data: :return: """ header_dict = {} data = str(data, encoding='utf-8') header, body = data.split('\r\n\r\n', 1) header_list = header.split('\r\n') for i in range(0, len(header_list)): if i == 0: if len(header_list[i].split(' ')) == 3: header_dict['method'], header_dict['url'], header_dict['protocol'] = header_list[i].split(' ') else: k, v = header_list[i].split(':', 1) header_dict[k] = v.strip() return header_dict def send_msg(conn, msg_bytes): """ WebSocket服务端向客户端发送消息 :param conn: 客户端连接到服务器端的socket对象,即： conn,address = socket.accept() :param msg_bytes: 向客户端发送的字节 :return: """ import struct token = b"\x81" length = len(msg_bytes) if length < 126: token += struct.pack("B", length) elif length <= 0xFFFF: token += struct.pack("!BH", 126, length) else: token += struct.pack("!BQ", 127, length) msg = token + msg_bytes conn.send(msg) return True def run(): sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock.bind(('127.0.0.1', 8003)) sock.listen(5) conn, address = sock.accept() data = conn.recv(1024) headers = get_headers(data) response_tpl = "HTTP/1.1 101 Switching Protocols\r\n" \ "Upgrade:websocket\r\n" \ "Connection:Upgrade\r\n" \ "Sec-WebSocket-Accept:%s\r\n" \ "WebSocket-Location:ws://%s%s\r\n\r\n" value = headers['Sec-WebSocket-Key'] + '258EAFA5-E914-47DA-95CA-C5AB0DC85B11' ac = base64.b64encode(hashlib.sha1(value.encode('utf-8')).digest()) response_str = response_tpl % (ac.decode('utf-8'), headers['Host'], headers['url']) conn.send(bytes(response_str, encoding='utf-8')) while True: try: info = conn.recv(8096) except Exception as e: info = None if not info: break payload_len = info[1] & 127 if payload_len == 126: extend_payload_len = info[2:4] mask = info[4:8] decoded = info[8:] elif payload_len == 127: extend_payload_len = info[2:10] mask = info[10:14] decoded = info[14:] else: extend_payload_len = None mask = info[2:6] decoded = info[6:] bytes_list = bytearray() for i in range(len(decoded)): chunk = decoded[i] ^ mask[i % 4] bytes_list.append(chunk) body = str(bytes_list, encoding='utf-8') send_msg(conn, body.encode('utf-8')) sock.close() if __name__ == '__main__': run()
websocket client(create_connection版)
''' on_ 为事件回调函数 ''' # coding:utf-8 import websocket try: import thread except ImportError: import _thread as thread import time def on_message(ws, message): '收到消息时' print(message) def on_error(ws, error): '发生错误时' print(error) def on_close(ws): '关闭连接时' print("### closed ###") def on_open(ws): '刚连接时' def run(*args): for i in range(3): time.sleep(1) ws.send("Hello Websocket %d" % i) time.sleep(1) ws.close() print("thread terminating...") thread.start_new_thread(run, ()) if __name__ == "__main__": websocket.enableTrace(True) ws = websocket.WebSocketApp("ws://127.0.0.1:8003", on_message=on_message, on_error=on_error, on_close=on_close) ws.on_open = on_open ws.run_forever()
第三方websocket server
为了方便测试，有的编码弱的同学可以使用第三方websocket server来测试（比如测试websocket测试网站、带websocket接口的网站(bitmex okex等)），这样你负责websocket client就好了
http://coolaf.com/tool/chattest
登录这个地址，手动测试一下，正常的话可以八这个ws:xxx地址复制到代码中测试
# coding:utf-8 from websocket import create_connection ws = create_connection("ws://123.207.167.163:9010/ajaxchattest") print("Sending 'Hello, World'...") ws.send("Hello, World") print("Sent") print("Receiving...") result = ws.recv() print("Received '%s'" % result) ws.close() websocket client(create_connection版)
# coding:utf-8 import websocket ws = websocket.WebSocket() ws.connect("ws://123.207.167.163:9010/ajaxchattest") ws.send("Hello, World") data = ws.recv() print(data) ws.close()
websocket client (WebSocketApp版)
1641
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
