hive导入 nginx 或 apache 日志

最新推荐文章于 2022-06-18 20:30:41 发布

架构师思考实践

最新推荐文章于 2022-06-18 20:30:41 发布

阅读量1.6k

点赞数 1

分类专栏： Hive 文章标签： hive导入 nginx 或 apach

本文链接：https://blog.csdn.net/smile0198/article/details/23750909

版权

Hive 专栏收录该内容

15 篇文章 0 订阅

订阅专栏

1、创建表

CREATE TABLE nginxlog (
 ipaddress STRING,
 identity STRING,
 user STRING,
 time STRING,
 request STRING,
 protocol STRING,
 status STRING,
 size STRING,
 referer STRING,
 agent STRING) 
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' 
WITH SERDEPROPERTIES (
 "input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*) ([^ ]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\".*\") ([^ \"]*|\".*\"))?","output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s %10$s")
STORED AS TEXTFILE;

2、加载日志数据

load data local inpath '/home/hadoop/anjl/access.log' overwrite into table nginxlog

3、查询

select * from nginxlog;

日志例子：

125.75.93.233 - - [14/Apr/2014:17:18:21 +0800] "GET /test/123.htm?a=1 HTTP/1.1" 200 14498 "-" -"Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
218.201.111.114 - - [14/Apr/2014:17:18:21 +0800] "GET /test/123.htm?a=1 HTTP/1.1" 200 17593 "-" -"Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
119.180.35.121 - - [14/Apr/2014:17:18:22 +0800] "GET /test/123.htm?a=1 HTTP/1.1" 200 19492 "-" -"Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"