这两天闲来无事,想学习下python,于是就把自己项目中用shell写的一个统计pv的模块改成了python,速度还不错,与大家分享下。
我读的一个3.2G大小的日志文件,格式如下:
2011-01-10 10:22:48,787 INFO [view] - am=adView<|>at=2011-01-10 10:22:48<|>aditemid=27<|>adposition=2<|>auid<|>ab=100123<|>targetab=100040<|>adid=*******<|>temp1=<|>temp2=<|>ip=<|>asid<|>cid=<|>csid=<|>useract=<|>ua=iPhone Simulator/4.2<|>flat=iphone<|>version=1.0<|>sm=iPhone Simulator<|>sv=4.2<|>language=en_US<|>ack=<|>location=<|>tag=<|>serverver=1.0.0<|>md5value=******<|>timestamp=2011-01-10 02:07:32 +0000<|>reserved=reserved<|>art=2<|>acode=000000
我统计的规则是统计相同的at(日期只考虑年月日,例如2011-01-10),aditemid,adposition各出现多少次,即如果at,aditemid,adposition在日志中出现过一次就记其一次pv。
python代码如下:
#!/usr/bin/python
#coding=utf-8
import re, time, datetime
import MySQLdb
startTime = datetime.datetime.now()
print startTime
conn = MySQLdb.connect(host='localhost', user='root', passwd='root', db='test', charset='utf8')
cursor = conn.cursor()
countMap = {}
sql = "insert into ap_report(ad_date, ad_app_id, view_position, view_pv) values(%s, %s, %s, %s) on duplicate key update view_pv=%s"
for row in open("D:/my documents/view.log","r"):
s = row.split('\n')[0].split("<|>")
# 唯一key main_id, pair_id
uniqueKey = '%s,%s,%s' % (s[1].split("=")[1].split(" ")[0], s[2].split("=")[1], s[3].split("=")[1])
if not countMap.has_key(uniqueKey):
countMap[uniqueKey] = '%s,%s,%s,1' % (s[1].split("=")[1].split(" ")[0], s[2].split("=")[1], s[3].split("=")[1])
else:
ad_date, ad_app_id, view_position, view_pv = countMap[uniqueKey].split(',')
view_pv = int(view_pv) + 1
countMap[uniqueKey] = '%s,%s,%s,%s' % (ad_date, ad_app_id, view_position, view_pv)
#入库
for key in countMap.keys():
ad_date, ad_app_id, view_position, view_pv = countMap[key].split(',')
param = (ad_date, ad_app_id, view_position, view_pv, view_pv)
cursor.execute(sql, param)
conn.commit()
cursor.close()
conn.close();
endTime = datetime.datetime.now()
print endTime
print u'用时:' + str(endTime - startTime)
print 'import complete'
运行结果:我机器是酷睿2的cpu(E5500),2g内存,在运行过程中cpu使用率在30%左右,还开着不少其他东西。。
2011-01-28 18:53:09.671000
2011-01-28 18:55:15.031000
用时:0:02:05.360000
import complete
跑了2分钟05秒,嘿嘿,速度还是不错的~~