企业项目实战---ELK日志分析平台之logstash数据采集（二）

最新推荐文章于 2024-06-02 02:45:00 发布

张一不二

最新推荐文章于 2024-06-02 02:45:00 发布

阅读量615

点赞数 1

分类专栏： ELK日志分析平台 linux学习文章标签： linux

本文链接：https://blog.csdn.net/qq_42003848/article/details/119707077

版权

linux学习同时被 2 个专栏收录

87 篇文章 4 订阅

订阅专栏

ELK日志分析平台

3 篇文章 2 订阅

订阅专栏

ELK日志分析平台

logstash数据采集

logstash数据采集

1.logstash简介

Logstash是一个开源的服务器端数据处理管道。

logstash拥有200多个插件,能够同时从多个来源采集数据,转换数据,然后将数据发送到您最喜欢的 “存储库” 中。(大多都是 Elasticsearch。)

Logstash管道有两个必需的元素,输入和输出,以及一个可选元素过滤器。
在这里插入图片描述

输入:采集各种样式、大小和来源的数据

Logstash 支持各种输入选择 ,同时从众多常用来源捕捉事件。
能够以连续的流式传输方式,轻松地从您的日志、指标、Web 应用、数据存储以及各种 AWS 服务采集数据。

过滤器:实时解析和转换数据

数据从源传输到存储库的过程中,Logstash 过滤器能够解析各个事件,识别已命名的字段以构建结构,并将它们转换成通用格式,以便更轻松、更快速地分析和实现商业价值。
- 利用 Grok 从非结构化数据中派生出结构
- 从 IP 地址破译出地理坐标
- 将 PII 数据匿名化,完全排除敏感字段
- 简化整体处理,不受数据源、格式或架构的影响

输出:选择您的存储库,导出您的数据

尽管 Elasticsearch 是我们的首选输出方向,能够为我们的搜索和分析带来无限可能,但它并非唯一选择。
Logstash 提供众多输出选择,您可以将数据发送到您要指定的地方,并且能够灵活地解锁众多下游用例

2.Logstash安装

logstash安装

[root@server6 ~]# ls
anaconda-ks.cfg  jdk-8u171-linux-x64.rpm  logstash-7.6.1.rpm
[root@server6 ~]# rpm -ivh jdk-8u171-linux-x64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:jdk1.8-2000:1.8.0_171-fcs        ################################# [100%]
Unpacking JAR files...
	tools.jar...
	plugin.jar...
	javaws.jar...
	deploy.jar...
	rt.jar...
	jsse.jar...
	charsets.jar...
	localedata.jar...
[root@server6 ~]# rpm -ivh logstash-7.6.1.rpm
warning: logstash-7.6.1.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:logstash-1:7.6.1-1               ################################# [100%]
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.30/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash

logstash命令

[root@server6 ~]# cd /usr/share/logstash/
[root@server6 logstash]# ls
bin           Gemfile       LICENSE.txt               modules     vendor
CONTRIBUTORS  Gemfile.lock  logstash-core             NOTICE.TXT  x-pack
data          lib           logstash-core-plugin-api  tools
[root@server6 logstash]# cd bin/
[root@server6 bin]# ls
benchmark.sh         logstash               logstash.lib.sh      pqrepair
cpdump               logstash.bat           logstash-plugin      ruby
dependencies-report  logstash-keystore      logstash-plugin.bat  setup.bat
ingest-convert.sh    logstash-keystore.bat  pqcheck              system-install
[root@server6 bin]# pwd
/usr/share/logstash/bin
[root@server6 bin]# /usr/share/logstash/bin/logstash 
benchmark.sh           logstash-keystore      pqrepair
cpdump                 logstash-keystore.bat  ruby
dependencies-report    logstash.lib.sh        setup.bat
ingest-convert.sh      logstash-plugin        system-install
logstash               logstash-plugin.bat    
logstash.bat           pqcheck

3.标准输入到标准输出

 /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'

[root@server6 bin]# /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-08-13 22:18:22.870 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2021-08-13 22:18:22.898 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2021-08-13 22:18:23.483 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-08-13 22:18:23.500 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.1"}
[INFO ] 2021-08-13 22:18:23.542 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"3ef2b8c0-2d53-4d04-b99a-e82f2699855f", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2021-08-13 22:18:25.724 [Converge PipelineAction::Create<main>] Reflections - Reflections took 55 ms to scan 1 urls, producing 20 keys and 40 values 
[WARN ] 2021-08-13 22:18:27.485 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2021-08-13 22:18:27.487 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x5b034ce4 run>"}
[INFO ] 2021-08-13 22:18:28.949 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-08-13 22:18:28.989 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
The stdin plugin is now waiting for input:
[INFO ] 2021-08-13 22:18:29.573 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
hello world
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
      "@version" => "1",
          "host" => "server6",
       "message" => "hello world",
    "@timestamp" => 2021-08-14T02:18:47.164Z
}
lalal
{
      "@version" => "1",
          "host" => "server6",
       "message" => "lalal",
    "@timestamp" => 2021-08-14T02:18:51.972Z
}
^C[WARN ] 2021-08-13 22:18:55.198 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2021-08-13 22:18:55.457 [Converge PipelineAction::Stop<main>] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2021-08-13 22:18:55.555 [LogStash::Runner] runner - Logstash shut down.

4.标准输入到文件

执行文本，指定输出位置/tmp/testfile和输出格式

[root@server6 bin]# cat test.conf 
input {
	stdin {}
}

output {
 file {
   path => "/tmp/testfile"
   codec => line { format => "custom format: %{message}"}
 }
}

[root@server6 bin]# /usr/share/logstash/bin/logstash -f test.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2021-08-13 22:20:32.191 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-08-13 22:20:32.212 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.1"}
[INFO ] 2021-08-13 22:20:34.637 [Converge PipelineAction::Create<main>] Reflections - Reflections took 69 ms to scan 1 urls, producing 20 keys and 40 values 
[WARN ] 2021-08-13 22:20:35.445 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2021-08-13 22:20:35.467 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/bin/test.conf"], :thread=>"#<Thread:0x1126f599 run>"}
[INFO ] 2021-08-13 22:20:36.655 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-08-13 22:20:36.716 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
The stdin plugin is now waiting for input:
[INFO ] 2021-08-13 22:20:37.080 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
redhat
[INFO ] 2021-08-13 22:20:46.481 [[main]>worker0] file - Opening file {:path=>"/tmp/testfile"}
zhangyi
[INFO ] 2021-08-13 22:21:05.421 [[main]>worker0] file - Closing file /tmp/testfile
^C[WARN ] 2021-08-13 22:21:06.948 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2021-08-13 22:21:07.192 [Converge PipelineAction::Stop<main>] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2021-08-13 22:21:07.476 [LogStash::Runner] runner - Logstash shut down.

查看输出文件

[root@server6 bin]# cat /tmp/testfile 
custom format: redhat
custom format: zhangyi

5.标准输入到es主机

文本内容

[root@server6 bin]# cat es.conf 
input {
	stdin {}
}

output {
	stdout {}

	elasticsearch {
		hosts => ["172.25.3.3:9200"]    #输出到的ES主机与端口
		index => "logstash-%{+yyyy.MM.dd}"   #定制索引名称
	}
}

执行文件

[root@server6 bin]# /usr/share/logstash/bin/logstash -f es.conf 
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2021-08-13 22:23:02.202 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-08-13 22:23:02.210 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.1"}
[INFO ] 2021-08-13 22:23:04.610 [Converge PipelineAction::Create<main>] Reflections - Reflections took 66 ms to scan 1 urls, producing 20 keys and 40 values 
[INFO ] 2021-08-13 22:23:06.522 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://172.25.3.3:9200/]}}
[WARN ] 2021-08-13 22:23:06.842 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://172.25.3.3:9200/"}
[INFO ] 2021-08-13 22:23:07.153 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2021-08-13 22:23:07.155 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2021-08-13 22:23:07.328 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//172.25.3.3:9200"]}
[INFO ] 2021-08-13 22:23:07.459 [Ruby-0-Thread-6: :1] elasticsearch - Using default mapping template
[WARN ] 2021-08-13 22:23:07.485 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2021-08-13 22:23:07.512 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/bin/es.conf"], :thread=>"#<Thread:0x798305c5 run>"}
[INFO ] 2021-08-13 22:23:07.569 [Ruby-0-Thread-6: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[INFO ] 2021-08-13 22:23:07.637 [Ruby-0-Thread-6: :1] elasticsearch - Installing elasticsearch template to _template/logstash
[INFO ] 2021-08-13 22:23:09.064 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2021-08-13 22:23:09.122 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
The stdin plugin is now waiting for input:
[INFO ] 2021-08-13 22:23:09.518 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
hello
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
          "host" => "server6",
      "@version" => "1",
       "message" => "hello",
    "@timestamp" => 2021-08-14T02:23:14.157Z
}
zhangyi
{
          "host" => "server6",
      "@version" => "1",
       "message" => "zhangyi",
    "@timestamp" => 2021-08-14T02:23:18.376Z
}
^C[WARN ] 2021-08-13 22:24:26.428 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2021-08-13 22:24:26.707 [Converge PipelineAction::Stop<main>] javapipeline - Pipeline terminated {"pipeline.id"=>"main"}
[INFO ] 2021-08-13 22:24:27.415 [LogStash::Runner] runner - Logstash shut down.

head插件内查看输出
在这里插入图片描述

在这里插入图片描述

6.指定文件输入到es主机

指定输入文件/var/log/messages，输出到172.25.3.3:9200

/usr/share/logstash/bin/logstash -f  /etc/logstash/conf.d/file-es.conf

[root@server6 conf.d]# cat file-es.conf 
input {
	file {
               path => "/var/log/messages"
               start_position => "beginning"
      }
}

output {
	stdout {}

	elasticsearch {
		hosts => ["172.25.3.3:9200"]
		index => "logstash-%{+yyyy.MM.dd}"
	}
}

查看输出内容
在这里插入图片描述

7.sincedb

logstash如何区分设备、文件名、文件的不同版本

logstash会把进度保存到sincedb文件中
想要从头重新输入，需要删除sincedb

# find / -name .sincedb*
/usr/share/logstash/data/plugins/inputs/file/.sincedb_45290
5a167cf4509fd08acb964fdb20c
# cd /usr/share/logstash/data/plugins/inputs/file/
# cat .sincedb_452905a167cf4509fd08acb964fdb20c
20297 0 64768 119226 1551859343.6468308
/var/log/messages
# ls -i /var/log/messages
20297 /var/log/messages

sincedb文件内容解释

 # cat .sincedb_452905a167cf4509fd08acb964fdb20c
20297 0 64768 119226 1551859343.6468308   /var/log/messages

sincedb文件一共6个字段

inode编号
文件系统的主要设备号
文件系统的次要设备号
文件中的当前字节偏移量
最后一个活动时间戳(浮点数)
与此记录匹配的最后一个已知路径

8.远程日志输入

logstash可以伪装成日志服务器,直接接受远程日志

配置server3/4，开放514端口，指向172.25.3.6

$ModLoad imudp
$UDPServerRun 514
*.* @@172.25.3.6:514

systemctl  restart  rsyslog.service

查看514端口

[root@server6 ~]# netstat  -antlp|grep :514
tcp6       0      0 :::514                  :::*                    LISTEN      21934/java          
tcp6       0      0 172.25.3.6:514          172.25.3.4:43182        ESTABLISHED 21934/java          
tcp6       0      0 172.25.3.6:514          172.25.3.3:54166        ESTABLISHED 21934/java

log.conf 指定syslog输入

[root@server6 conf.d]# cat log.conf 
input {
         syslog {

            port => 514
    }
}

output {
	stdout {}

	elasticsearch {
		hosts => ["172.25.3.3:9200"]
		index => "syslog-%{+yyyy.MM.dd}"
	}
}

执行文件，查看syslog
在这里插入图片描述

9.多行过滤插件

输入信息一般一行为一段内容，但对于错误信息，需要多行归于一个日志信息

[2021-08-14T00:53:42,821][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [server3] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Failed to parse value [fals] as only [true] or [false] are allowed.
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.1.jar:7.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.1.jar:7.6.1]
Caused by: java.lang.IllegalArgumentException: Failed to parse value [fals] as only [true] or [false] are allowed.
	at org.elasticsearch.common.Booleans.parseBoolean(Booleans.java:73) ~[elasticsearch-core-7.6.1.jar:7.6.1]
	at org.elasticsearch.common.settings.Setting.parseBoolean(Setting.java:1279) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.common.settings.Setting.lambda$boolSetting$24(Setting.java:1256) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:433) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.common.settings.Setting.get(Setting.java:427) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.cluster.node.DiscoveryNode.isDataNode(DiscoveryNode.java:69) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:321) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.1.jar:7.6.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.1.jar:7.6.1]

单行输入展示

[root@server6 conf.d]# cat es-log.conf 
input {
      file {
        path => "/var/log/my-es.log"
        start_position => "beginning"
          #codec => multiline {
            #pattern => "^\["
            #negate => "true"
            #what => "previous"
          #}
      }
}

output {
	stdout {}

	elasticsearch {
		hosts => ["172.25.3.3:9200"]
		index => "eslog-%{+yyyy.MM.dd}"
	}

}

 /usr/share/logstash/bin/logstash -f es-log.conf

在这里插入图片描述

添加多行过滤，再次执行

[root@server6 conf.d]# cat es-log.conf 
input {
      file {
        path => "/var/log/my-es.log"
        start_position => "beginning"
          codec => multiline {
            pattern => "^\["
            negate => "true"
            what => "previous"
          }
      }
}

output {
	stdout {}

	elasticsearch {
		hosts => ["172.25.3.3:9200"]
		index => "eslog-%{+yyyy.MM.dd}"
	}

}

在执行前要删除sincedb，重新录入


 
 [root@server6 file]# ls
[root@server6 file]# ls -a
.  ..  .sincedb_13f094911fdac7ab3fa6f4c93fee6639
[root@server6 file]# rm -rf .sincedb_13f094911fdac7ab3fa6f4c93fee6639
[root@server6 file]# pwd
/usr/share/logstash/data/plugins/inputs/file

 /usr/share/logstash/bin/logstash -f es-log.conf

查看多行过滤效果
在这里插入图片描述

10.grok过滤

grok过滤文本

[root@server6 conf.d]# cat grok.conf 
input {
	stdin {}
}

filter {
	grok {
	match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
	}
}

output {
	stdout {}
}

执行grok.conf

 /usr/share/logstash/bin/logstash -f grok.conf

输入内容，查看切片信息

55.3.244.1 GET /index.html 15824 0.043

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
       "message" => "55.3.244.1 GET /index.html 15824 0.043",
        "client" => "55.3.244.1",
        "method" => "GET",
          "host" => "server6",
    "@timestamp" => 2021-08-14T05:53:23.078Z,
      "duration" => "0.043",
      "@version" => "1",
       "request" => "/index.html",
         "bytes" => "15824"
}

11.apache服务日志过滤实战

grok http过滤模块

[root@server6 conf.d]# cd /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns
[root@server6 patterns]# ls
aws     bind  exim       grok-patterns  httpd  junos         maven        mcollective-patterns  nagios      rails  ruby
bacula  bro   firewalls  haproxy        java   linux-syslog  mcollective  mongodb               postgresql  redis  squid
[root@server6 patterns]# vim httpd 
[root@server6 patterns]# cat httpd 
HTTPDUSER %{EMAILADDRESS}|%{USER}
HTTPDERROR_DATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}

# Log formats
HTTPD_COMMONLOG %{IPORHOST:clientip} %{HTTPDUSER:ident} %{HTTPDUSER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)
HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} %{QS:referrer} %{QS:agent}

# Error logs
HTTPD20_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{LOGLEVEL:loglevel}\] (?:\[client %{IPORHOST:clientip}\] ){0,1}%{GREEDYDATA:message}
HTTPD24_ERRORLOG \[%{HTTPDERROR_DATE:timestamp}\] \[%{WORD:module}:%{LOGLEVEL:loglevel}\] \[pid %{POSINT:pid}(:tid %{NUMBER:tid})?\]( \(%{POSINT:proxy_errorcode}\)%{DATA:proxy_message}:)?( \[client %{IPORHOST:clientip}:%{POSINT:clientport}\])?( %{DATA:errorcode}:)? %{GREEDYDATA:message}
HTTPD_ERRORLOG %{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}

# Deprecated
COMMONAPACHELOG %{HTTPD_COMMONLOG}
COMBINEDAPACHELOG %{HTTPD_COMBINEDLOG}

查看http access_log日志内容

[root@server6 conf.d]# cat /var/log/httpd/access_log 
172.25.3.6 - - [14/Aug/2021:01:56:11 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"
172.25.3.6 - - [14/Aug/2021:01:56:12 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"
172.25.3.6 - - [14/Aug/2021:01:56:13 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"
172.25.3.6 - - [14/Aug/2021:01:56:27 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"
172.25.3.6 - - [14/Aug/2021:01:56:28 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"
172.25.3.6 - - [14/Aug/2021:01:56:28 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"
172.25.3.6 - - [14/Aug/2021:01:56:43 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"
172.25.3.6 - - [14/Aug/2021:01:56:48 -0400] "GET / HTTP/1.1" 200 11 "-" "curl/7.29.0"

为目录添加755权限

[root@server6 patterns]# chmod 755 /var/log/httpd/
[root@server6 patterns]#  ll -d  /var/log/httpd
drwxr-xr-x 2 root root 41 Aug 14 01:55 /var/log/httpd

apache.conf 内容

[root@server6 conf.d]# cat apache.conf 
input {
	file {
		path => "/var/log/httpd/access_log"
		start_position => "beginning"
	}

}

filter {
	grok {
	match => { "message" => "%{HTTPD_COMBINEDLOG}" }
	}
}

output {
	stdout {}

	elasticsearch {
		hosts => ["172.25.0.3:9200"]
		index => "apachelog-%{+yyyy.MM.dd}"
	}
	
}

执行

 /usr/share/logstash/bin/logstash -f apache.conf

查看切片内容

{
          "agent" => "\"curl/7.29.0\"",
           "auth" => "-",
           "host" => "server6",
       "@version" => "1",
       "referrer" => "\"-\"",
    "httpversion" => "1.1",
          "ident" => "-",
          "bytes" => "11",
           "verb" => "GET",
     "@timestamp" => 2021-08-14T06:08:50.340Z,
           "path" => "/var/log/httpd/access_log",
        "request" => "/",
        "message" => "172.25.3.6 - - [14/Aug/2021:01:56:48 -0400] \"GET / HTTP/1.1\" 200 11 \"-\" \"curl/7.29.0\"",
       "clientip" => "172.25.3.6",
      "timestamp" => "14/Aug/2021:01:56:48 -0400",
       "response" => "200"
}