Graylog日志收集分析系统部署记录

一.框架描述

       框架使用rsylog+kafka+graylog+elasticsearch技术，其中rsyslog作为日志转发和分解工具，kafka作为日志暂存队列和缓存工具，graylog则作为日志流处理以及查询工具，elasticsearch作为日志索引和存储。

       首先，rsyslog可以通过不同的输入组件来抓取或接收日志，目前提供的并且有可能使用到的方式包括：tcp、udp、文件读取等方式，而输出则支持通过文件输出、kafka输出等，各种组件可参照官方文档。

       rsyslog通过从文件读取或者接收udp消息获取日志，进行一些脚本处理（对日志格式化，处理多行日志等），rsyslog利用omkafka组件把消息转发给kafka，同时graylog也支持多钟日志输入方式，包括http、tcp、udp等，当然也支持

kafka，于是在graylog创建一个kafka输入，从kafka中把rsyslog推入的消息消费出来，然后通过graylog的流处理把消息保存进入elasticsearch，以上是一个日志的收集和存储流程。

     日志的查询流程则是graylog通过调用elasticsearch查询elasticsearch中的日志，同时graylog和elasticsearch提供了很多Restful api可供我们使用。

     本章主要描述几个重点框架的搭建和部署，后面一篇会描述怎么跟docker项目结合起来，同时列出一些可能用到的graylog rest api。

二.框架安装部署

1.rsyslog安装部署

a.修改/etc/yum.repos.d/rsyslog.repo文件为以下内容，安装包（）

[rsyslog_v8] name=Adiscon CentOS-$releasever - local packages  for  $basearch baseurl=http: //rpms.adiscon.com/v8-stable/epel-$releasever/$basearch enabled= 1 gpgcheck= 0 gpgkey=http: //rpms.adiscon.com/RPM-GPG-KEY-Adiscon protect= 1

b.执行 yum install rsyslog

c.安装rsyslog的kafka组件 执行命令：yum install rsyslog-kafka.x86_64（kafka）

  input modules

  output moudles

d.修改/etc/rsyslog.conf 配置,配置文件参照下面，该脚本是抓取日志文件中的日志，通过omkafka发送出去，日志格式是GELF格式，同时支持多行处理，凡是日志按照yyyy-mm-dd hh:mm:ss 开头才算是一条新的日志 ，详细请参考官网的Rainer Script

  读取/etc/rsyslog.d/*.conf

# 加载组件 module(load= "imtcp" ) module(load= "omkafka" ) # tcp输入 input(type= "imtcp"        port= "10514"        ruleset= "writeRemoteData" ) # 租户json日志 template(name= "tenant_json"  type= "list" ) {      constant(value= "{" ) constant(value= "\"version\":\"1\"" ) constant(value= "," )      constant(value= "\"logtype\":\"tenant\"" ) constant(value= "," )      constant(value= "\"timestamp\":\"" )          property(name= "timereported"  dateFormat= "rfc3339"  format= "json" ) constant(value= "\"," )  //单条日志的变量      constant(value= "\"providerId\":\"" )         property(name= "$!provider_id"  format= "json" ) constant(value= "\"," )    //$！自定制变量，配置文件定义的      constant(value= "\"container\":\"" )         property(name= "$!container"  format= "json"  ) constant(value= "\"," )    //$无！号表示全局变量，myhostname , $. 和 $!      constant(value= "\"image\":\"" )             property(name= "$!image"  format= "json"  ) constant(value= "\"," )      constant(value= "\"host\":\"" )               property(name= "source"  format= "json"  ) constant(value= "\"," )    // $myhostname      constant(value= "\"severity\":\"" )           property(name= "syslogseverity-text"  format= "json" ) constant(value= "\"," )      constant(value= "\"facility\":\"" )           property(name= "syslogfacility-text"  format= "json" ) constant(value= "\"," )      constant(value= "\"programname\":\"" )        property(name= "programname"  format= "json" ) constant(value= "\"," )      constant(value= "\"procid\":\"" )             property(name= "procid"  format= "json" ) constant(value= "\"," )      constant(value= "\"message\":" )              property(name= "msg"  position.from= "2" ) constant(value= "}" ) constant(value= "\n" ) } # 租户text日志 template(name= "tenant_text"  type= "list" ) {      constant(value= "{" ) constant(value= "\"version\":\"1\"" ) constant(value= "," )      constant(value= "\"logtype\":\"tenant\"" ) constant(value= "," )      constant(value= "\"timestamp\":\"" )          property(name= "timereported"  dateFormat= "rfc3339"  format= "json" ) constant(value= "\"," )      constant(value= "\"providerId\":\"" )         property(name= "$!provider_id"  format= "json" ) constant(value= "\"," )      constant(value= "\"container\":\"" )         property(name= "$!container"  format= "json"  ) constant(value= "\"," )      constant(value= "\"image\":\"" )             property(name= "$!image"  format= "json"  ) constant(value= "\"," )      constant(value= "\"host\":\"" )               property(name= "source"  format= "json"  ) constant(value= "\"," )      constant(value= "\"severity\":\"" )           property(name= "syslogseverity-text"  format= "json" ) constant(value= "\"," )      constant(value= "\"facility\":\"" )           property(name= "syslogfacility-text"  format= "json" ) constant(value= "\"," )      constant(value= "\"programname\":\"" )        property(name= "programname"  format= "json" ) constant(value= "\"," )      constant(value= "\"procid\":\"" )             property(name= "procid"  format= "json" ) constant(value= "\"," )      constant(value= "\"message\":\"" )            property(name= "msg"  position.from= "2"  format= "json" ) constant(value=