Logstash 读取tomcat错误日志

27人阅读 评论(0) 收藏 举报
分类:

Logstash 读取tomcat错误日志

概述

最近搭建了elk日志分析系统、想读取一下tomcat的错误日志、但是一个异常由于换行总是分多次存储展示、导致不是很清晰的看一个错误日志信息

  • inputs 输入
  • codecs 解码
  • filters 过滤
  • outputs 输出

image image image

例子

安装插件 logstash-filter-multiline

在线安装插件

logstash-plugin.bat install logstash-filter-multiline

升级插件 
logstash-plugin.bat update logstash-filter-multiline

离线安装
logstash-plugin.bat install logstash-filter-multiline.gem

卸载
logstash-plugin.bat uninstall logstash-filter-multiline

例子

如果不是以 “[“开头的日志 都跟上一个日志合并在一起。以此类推遇到其他的多行日志也可以按照这个方法来做合并。

input {  
    file { 
		type => "tomcat_error"
        path => "E:/install/temp/*.log"
		start_position => beginning
    }  
} 

filter {
    if [type] == "tomcat_error" {
            multiline {
                      pattern => "^[^\[]"
                      what => "previous"
                  }
                mutate {
           split => ["message", "|"]
        }
        grok {
            match => { 
                           "message" => "(?m)%{TIMESTAMP_ISO8601:logtime}"
            }
        }
    }
}
 

output {    
    elasticsearch { hosts => localhost   
                    index => "logstash-cmis"  
                    document_type =>"tomcat"}  
      
    stdout { codec => rubydebug  }   
}  

例子一

input {  
    file {  
        path => "E:/install/temp/*.log"
		start_position => beginning
    }  
} 
 

output {    
    elasticsearch { hosts => localhost   
                    index => "logstash-cmis"  
                    document_type =>"tomcat"}  
      
    stdout { codec => json_lines }   
}  

例子二


input {  
    file {  
        path => "E:/install/temp/*.log"
		start_position => beginning
        codec => multiline {  
            pattern => "^\s"  
            what => "previous"  
        }  
    }  
} 

 filter {  
    multiline {    
		pattern => "^\s+%{TIMESTAMP_ISO8601}"  
		negate=>true    
		what=>"previous"    
	}

}

output {    
    elasticsearch { hosts => localhost   
                    index => "logstash-cmis"  
                    document_type =>"tomcat"}  
      
    stdout { codec => json_lines }   
}  

例子

input {  
    file{  
        path => "E:/install/temp/*.csv"  
        start_position => beginning  
    }  
 }    
    
filter {  
    grok {  
        patterns_dir => "patterns.txt"  
        match =>{ "message" => "%{DATA:name},%{DATA:person_id},%{DATA:email},%{DATA:tel},%{DATA:adress},%{DATA:from}%{S}" }  
        }  
    mutate{  
        remove_field => ["host","path","message","@version"]  
}  
}  
    
output {    
    elasticsearch { hosts => localhost   
                    index => "sgdb"  
                    document_type =>"sgdb"}  
      
    stdout { codec => rubydebug }   
}  

例子四

  • 分割匹配日志

日志格式

2015-09-28·09:50:48·[http-bio-80-exec-13]·DEBUG·com.weitoo.server.aspect.LogAspect·-{ip:183.16.4.40,url:http://api.xx.com/server/sc/commodity/getOnlineCommodity,param:{"shopId":1000001,"needCategory":false,"needCommodityTotal":false,"searchCommodityId":1002001},return:{"status":1},cost:3.911ms}
SERVER_LOG %{DATA:year}-%{DATA:month}-%{DATA:day}\ %{DATA:hour}\:%{DATA:min}\:%{DATA:sec}\ %{DATA:level}\ %{DATA:class} -{ip:%{DATA:ip},url:%{DATA:url},param:%{DATA:param},return:%{DATA:return},cost:%{BASE10NUM:cost}

conf

input {
  file {
   type=>"xx_server_log"
   path=>"/opt/software/apache-tomcat-7.0.59/logs/catalina.out"
   codec=> multiline {
           pattern => "(^.+Exception:.+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
           what=> "previous"
    }

 }
}



filter {
        if [type] == "xx_server_log" {
           grok {
                 match => [ "message","%{SERVER_LOG}"]
                 patterns_dir => ["/opt/conf/logstash"]
                 remove_field => ["message"]
          }
        }
}


output {
   elasticsearch {
  host =>"xx-management"
  protocol =>"http"
  workers => 5
  template_overwrite => true

}
   stdout { codec=> rubydebug }
}

清空es数据

DELETE /logstash-cmis

DELETE /.kibana/index-pattern/logstash-cmis

GET /_search
查看评论

logstash 分析nginx 错误日志

[root@dr-mysql01 frontend-error]# cat logstash_error.conf input { file { ty...
  • zhaoyangjian724
  • zhaoyangjian724
  • 2016-08-24 16:43:09
  • 1593

logstash 收集tomcat log demo

input{ file { path => "D:/server/apache-tomcat-8.0.15-windows-2014/apache-tomcat-8.0.15/log...
  • earbao
  • earbao
  • 2015-10-26 15:45:15
  • 5668

logstash 处理tomcat catalina.out

input { file { type => "zj_api" path => ["/data01/applog_bac...
  • zhaoyangjian724
  • zhaoyangjian724
  • 2016-10-13 13:17:25
  • 2328

logstash 处理tomcat日志

[root@dr-mysql01 tomcat]# cat logstash_tomcat.conf input { file { type => "...
  • zhaoyangjian724
  • zhaoyangjian724
  • 2016-08-29 20:08:03
  • 4737

logstash tomcat catalina.out 告警

[elk@dr-mysql01 tomcat]$ cat logstash_tomcat.conf input { file { type => "zj...
  • zhaoyangjian724
  • zhaoyangjian724
  • 2016-09-21 17:02:06
  • 1274

使用logstash配置nginx和tomcat日志统一收集到一台日志服务器

相关服务器安装jdk和redis请参考 http://blog.csdn.net/u013619834/article/details/38894649 http://blog.csdn.net/u0...
  • u013619834
  • u013619834
  • 2016-11-18 19:15:56
  • 3739

tomcat-juli.jar

  • 2009年03月17日 22:39
  • 59KB
  • 下载

ELK实战-Logstash multiline:识别错误堆栈

概述在通过ELK收集日志的时候,通常需要对日志进行分析,例如实时监控错误堆栈,并进行告警。通常错误堆栈都是多行的,但通常ELK默认都是识别单行的,怎么才能多行呢?logstash的codec、filt...
  • K_Zombie
  • K_Zombie
  • 2016-04-15 00:14:50
  • 4877

logstash 处理nginx 错误日志

2016/08/30 14:52:02 [error] 11325#0: *346 open() "/var/www/zjzc-web-frontEnd/%27%22%2f%3E%3C%2fscrip...
  • zhaoyangjian724
  • zhaoyangjian724
  • 2016-08-30 15:00:23
  • 1464

ELK安装和配置,简单演示监控tomcat日志记录

ELK的安装和配置,简单演示监控tomcat日志记录 1.安装,地址      官网:https://www.elastic.co/      logstash:https://www.elast...
  • yipanbo
  • yipanbo
  • 2017-02-07 19:08:33
  • 5578
    我的微信
      我的微信号号,添加后更多福利
    个人资料
    持之以恒
    等级:
    访问量: 6万+
    积分: 4598
    排名: 7991
    最新评论