使用python解析apache日志并上传到HBase_python apache 日志解析-CSDN博客

本文链接：https://blog.csdn.net/lotusdeng2010/article/details/7004537

本文介绍了如何利用Thrift和HBase整合Apache日志文件，通过创建表、插入记录来处理日志数据，并对处理过程进行了性能测试，包括解析速度和上传至HBase的时间成本。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

环境设置：

安装thrift, 默认情况下thrift python包在/usr/python2.7/site-packages/thrift下
安装apachelog python包
运行thrift --gen py HBase.thrift (在{HBASE_HOME}/src/java/org/apache/hadoop/hbase/thrift目录下),会在当前目录下产生gen-py目录，里面包含了hbase的python包
在eclipse里新建python项目，设置项目的属性，在PyDev-PYTHONPATH属性页中，ExternalLibraries Tab页中，添加sourcefolder /home/lotus/work/hbase-0.90.4-cdh3u2/gen-py;/usr/lib/python2.7/site-packages

import sys
import time
from thrift import Thrift  
from thrift.transport import TSocket, TTransport  
from thrift.protocol import TBinaryProtocol  
from hbase import ttypes  
from hbase.Hbase import *  
import apachelog



# tableName apachefile1, apachefile2, ..
if __name__ == "__main__":
    
    if(len(sys.argv) < 2):
        print "command line: tableName apachefile1, apachefile2, ...."
        print sys.argv
    
    tableName = sys.argv[1];
    transport = TSocket.TSocket('namenode0', 9090)
    transport = TTransport.TBufferedTransport(transport)
    protocol = TBinaryProtocol.TBinaryProtocol(transport)
    client = Client(protocol)
    transport.open()
    # create table
    column1 = ColumnDescriptor(name = "cf1:ipaddress")
    column2 = ColumnDescriptor(name = "cf1:finishtime")
    column3 = ColumnDescriptor(name = "cf1:requestline")
    column4 = ColumnDescriptor(name = "cf1:returncode")
    column5 = ColumnDescriptor(name = "cf1:size")
    column6 = ColumnDescriptor(name = "cf1:url")
    column7 = ColumnDescriptor(name = "cf1:browser")
    
    try:
        client.createTable(tableName, [column1, column2, column3, column4, column5, column6, column7])
    except Exception, e:
        print "create table fail:" + str(e)
        
    #insert apache record
    startTime = time.time()
    
    apacheParser = apachelog.parser(r'%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"')
    current_row = 0
    for i in range(2, len(sys.argv)):
        file = sys.argv[i]
        print "handle file:", file
        file = open(file, "r");
        for line in file.readlines():
            data = apacheParser.parse(line)
            
            ipaddress = data['%h']
            finishTime = data['%t']
            requestLine = data['%r']
            returnCode = data['%>s']
            size = data['%l']
            url = data['%u']
            browser = data['%{User-Agent}i']
            
            mutations = [Mutation(column = "cf1:ipaddress", value = ipaddress),
                         Mutation(column = "cf1:finishTime", value = finishTime),
                         Mutation(column = "cf1:requestLine", value = requestLine),
                         Mutation(column = "cf1:returnCode", value = returnCode),
                         Mutation(column = "cf1:size", value = size),
                         Mutation(column = "cf1:url", value = url),
                         Mutation(column = "cf1:browser", value = browser)]
            
            client.mutateRow(tableName, str(current_row), mutations)
            print "insert row"
            current_row = current_row + 1
            
    
    endTime = time.time()
         
    print "total insert rows:", str(current_row)
    print "cost time:" + str(endTime - startTime)
    print("exit")

性能测试：

处理一个大小为19.9MB，有84558行的Apache日志文件，解析并上传到Hbase共花了278秒