elk(1)-趁离职总结下我前公司elk方案

=============这篇文章主要书简介elk的学习方法了，建议直接跳到下节看================

趁离职有空，想起自己在前公司，博X科技，在那家公司也是误打误撞的进入的elk方案的搭建与使用。

elk是三个组件合成的一个方案。

elasticsearch+logstash+kibana

===========必须要看的书============

研究了也大概是一个月左右了，看了三本书，这三本书很重要，大家可以看看。

elasticsearch-the-definitive-guide-cn.pdf

kibana中文指南.pdf

Logash+Elasticsearch+Kibana 日志系统安装部署.pdf

============就说说能实现的效果吧==================

需求是这样的 有份这样的kafka日志

[2016-08-30 13:00:58,878] INFO Closing socket connection to /192.167.42.146. (kafka.network.Processor)
\[%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})\] %{WORD} %{WORD:operate} socket connection to /%{IP:cilent}. \((?<tip>.*)\)

[2016-08-30 13:00:59,343] ERROR Closing socket for /192.167.42.146 because of error (kafka.network.Processor)
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at kafka.utils.Utils$.read(Utils.scala:375)
        at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
        at kafka.network.Processor.read(SocketServer.scala:347)
        at kafka.network.Processor.run(SocketServer.scala:245)
        at java.lang.Thread.run(Thread.java:745)
		
[2016-08-30 13:01:26,143] INFO Rolled new log segment for 'CITY_SHARE_LTE_UP_GZ-37' in 4 ms. (kafka.log.Log)
[2016-08-30 13:02:15,251] INFO Scheduling log segment 5567051286 for log CITY_SHARE_LTE_UP_DG-33 for deletion. (kafka.log.Log)
[2016-08-30 13:03:15,196] INFO Deleting segment 23423790671 from log CITY_SHARE_DG-4. (kafka.log.Log)
[2016-08-30 13:03:15,368] INFO Deleting index /sidata12/kafka-logs/CITY_SHARE_LTE_UP_SZ-28/00000000009896455083.index.deleted (kafka.log.OffsetIndex)

要收集关键字，收集日志然后分析

%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND}
Aug 28 04:10:01 jumpserver sshd[57279]:



secure日志样本-------------------------------

crond的
Aug 30 02:50:01 jumpserver crond[23995]: pam_limits(crond:session): unknown limit item 'nsoft'   Y


sshd的
	
Aug 28 07:29:55 jumpserver sshd[54007]: pam_unix(sshd:session): session closed for user liujing //session登陆失败的
(?<logstime>%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND}) %{WORD} %{WORD}\[%{NUMBER:secureid}\]\: %{WOR
D:pam}\(%{WORD}\:%{WORD}\)\: %{WORD} %{WORD} %{WORD} %{WORD} %{WORD:user}

Aug 28 08:07:51 jumpserver sshd[55645]: pam_unix(sshd:session): session opened for user gdsignal by (uid=0)  //session登陆成功的
%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND} %{WORD} %{WORD}\[%{NUMBER:secureid}\]\: %{WORD:pam}\(%{WORD}\:%{WORD:sshd}
\)\: %{WORD} %{WORD} %{WORD} %{WORD} %{WORD:user} %{WORD} \(uid\=%{NUMBER:uid}\)

Aug 28 13:49:42 jumpserver sshd[57279]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.201.38.237  user=chencaixia //认证失败的
%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND} %{WORD} %{WORD}\[%{NUMBER:secureid}\]\: %{WORD:pam}\(%{WORD}\:%{WORD:sshd}\)\: %{WORD} %{WORD}\; logname\=(%{NUMBER:logname}| )uid\=%{NUMBE
R:uid} euid\=%{NUMBER:euid} tty\=%{WORD:tty} ruser\=(%{WORD:ruser}| )rhost\=%{IP:cilent}  user\=%{WORD:user}

Aug 28 08:07:51 jumpserver sshd[55645]: Accepted password for gdsignal from 10.201.38.237 port 17490 ssh2 //登陆成功
%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND} %{WORD} %{WORD}\[%{NUMBER}\]\: Accepted password fo
r %{WORD:user} from %{IP:cilent} port %{NUMBER:logport} %{WORD:tty}

Aug 28 13:56:32 jumpserver sshd[57392]: Received disconnect from 192.168.35.27: 11: disconnected by user
%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND} %{WORD} %{WORD}\[%{NUMBER}\]\: Received disconnect 
from %{IP:cilent}\: %{NUMBER:error_type}\: disconnected by %{WORD:user}

Aug 29 09:39:39 jumpserver sshd[65367]: Invalid user zengqin from 10.201.38.237
%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND} %{WORD} %{WORD}\[%{NUMBER}\]\: Invalid user %{WORD:user} from %{IP:cilent}

Aug 28 13:49:54 jumpserver sshd[57280]: Disconnecting: Too many authentication failures for chencaixia
%{MONTH} %{NUMBER} %{HOUR}:%{MINUTE}:%{SECOND} %{WORD} %{WORD}\[%{NUMBER}\]\: Disconnecting: Too many authentication failures for %{WORD:user}

Aug 28 13:57:01 jumpserver sshd[57409]: subsystem request for sftp

Aug 28 13:49:54 jumpserver sshd[57279]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=10.201.38.237  user=chencaixia
Aug 28 13:49:54 jumpserver sshd[57279]: PAM service(sshd) ignoring max retries; 6 > 3

Aug 28 14:10:59 jumpserver su: pam_tally2(su-l:auth): unknown option: no_magic_root   Y

根据样本的信息，写出的正则表达式，然后收集需要的日志logstash部分的

curl -XPUT http://172.16.248.24:9200/mymegacorp -d '
{
		
	"mappings" : {
		  "employee" : {
			"properties" : {
			  "fitstname" : {
				"type" : "string"
				"index" : "not_analyzed"
			  },
			  "interesrs" : {
				"type" : "string"
			  },
			  "lastname" : {
				"type" : "string"
			  }
			}
		  }
		}
  
}
'

写出相关的elasticsearch的索引put进去

最后就是kibana展示-----------------流程就是这样的 ，接下来各个组件详细介绍