python日志分析框架_利用Python中的pandas库对cdn日志进行分析详解

前言

最近工作工作中遇到一个需求，是要根据CDN日志过滤一些数据，例如流量、状态码统计，TOP IP、URL、UA、Referer等。以前都是用 bash shell 实现的，但是当日志量较大，日志文件数G、行数达数千万亿级时，通过 shell 处理有些力不从心，处理时间过长。于是研究了下Python pandas这个数据处理库的使用。一千万行日志，处理完成在40s左右。

代码

#!/usr/bin/python

# -*- coding: utf-8 -*-

# sudo pip install pandas

__author__ = 'Loya Chen'

import sys

import pandas as pd

from collections import OrderedDict

"""

Description: This script is used to analyse qiniu cdn log.

================================================================================

日志格式

IP - ResponseTime [time +0800] "Method URL HTTP/1.1" code size "referer" "UA"

================================================================================

日志示例

[0] [1][2] [3] [4] [5]

101.226.66.179 - 68 [16/Nov/2016:04:36:40 +0800] "GET http://www.qn.com/1.jpg -"

[6] [7] [8] [9]

200 502 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"

================================================================================

"""

if len(sys.argv) != 2:

print('Usage:', sys.argv[0], 'file_of_log')

exit()

else:

log_file = sys.argv[1]

# 需统计字段对应的日志位置

ip = 0

url = 5

status_code = 6

size = 7

referer = 8

ua = 9

# 将日志读入DataFrame

reader = pd.read_table(log_file, sep=' ', names=[i for i in range(10)], iterator=True)

loop = True

chunkSize = 10000000

chunks = []

while loop:

try:

chunk &#