Python+pandas对系统错误进行日志分析

Python+pandas对系统错误进行日志分析

生产日志一般都很大，即使就是一天有时也是很大的，如果想要对错误日志进行统计，在bash shell里处理相当费时，使用pandas处理，千万级别的日志，也不会超过1分钟。
我的生产日志是这样的：

现在要分别统计错误代码是0-9的错误次数，代码这样实现：

#coding=gbk
import pandas as pd
log=pd.read_table('my.log', sep='\n', iterator=True,encoding='gbk',header=None,names=['desc'])
loop = True
chunkSize = 10000
chunks = []
while loop:
    try:
        chunk = log.get_chunk(chunkSize)
        chunks.append(chunk)
    except StopIteration:
        loop=False
        
log=pd.concat(chunks,ignore_index=False)
print('错误 0='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 0',:].count()['desc'])))
print('错误 1='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 1',:].count()['desc'])))
print('错误 2='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 2',:].count()['desc'])))
print('错误 3='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 3',:].count()['desc'])))
print('错误 4='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 4',:].count()['desc'])))
print('错误 5='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 5',:].count()['desc'])))
print('错误 6='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 6',:].count()['desc'])))
print('错误 7='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 7',:].count()['desc'])))
print('错误 8='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 8',:].count()['desc'])))
print('错误 9='+str((log.loc[log['desc']=='com.sfit.mobile.core.AppAccessException: 9',:].count()['desc'])))

介绍下pd.read_table()，感兴趣可以研究下每个参数使用：

图解Python数据结构与算法-实战篇