微服务ELK分布式日志logstash7.15.1读取文件并写入elasticsearch7.5.1（基本配置）

elasticsearch7.5.1和kibana7.5.1以及logstash7.15.1的安装部署这里略过。

1. 基本配置文件（logstash2.conf）

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  file {
    start_position => end 
    path => "E:/home/wxp/box/task/box-task-info.log"
  }
 
}
 
output {
  stdout{}
  elasticsearch {
   #es地址，可多个
   hosts => ["localhost:9200"]
    action => "index"
	#获取输出参数"indexname"值当做索引，如果没有则会自动创建对应索引(需要es开启自动创建索引)
    index => "test_log_index"
   }
}

2. 启动logstash

logstash -f logstash2.conf

3. 添加文件内容，执行测试

在文件尾添加如下内容，然后保存文件。

17:09:37.168 [main] INFO  o.q.c.QuartzScheduler - [shutdown,740] - Scheduler RuoyiScheduler_$_DESKTOP-O93E7VQ1565255137077 shutdown complete.
17:09:37.170 [main] INFO  c.a.d.p.DruidDataSource - [close,1928] - {dataSource-1} closed
17:09:37.173 [main] INFO  o.a.c.c.StandardService - [log,173] - Stopping service [Tomcat]

可以看到logstash控制台输出

{
      "@version" => "1",
       "message" => "17:09:37.173 [main] INFO  o.a.c.c.StandardService - [log,173] - Stopping service [Tomcat]\r",
          "host" => "DESKTOP-O93E7VQ",
    "@timestamp" => 2022-02-19T02:36:36.338Z,
          "path" => "E:/home/wxp/box/task/box-task-info.log"
}
{
      "@version" => "1",
       "message" => "17:09:37.168 [main] INFO  o.q.c.QuartzScheduler - [shutdown,740] - Scheduler RuoyiScheduler_$_DESKTOP-O93E7VQ1565255137077 shutdown complete.\r",
          "host" => "DESKTOP-O93E7VQ",
    "@timestamp" => 2022-02-19T02:36:36.313Z,
          "path" => "E:/home/wxp/box/task/box-task-info.log"
}
{
      "@version" => "1",
       "message" => "17:09:37.170 [main] INFO  c.a.d.p.DruidDataSource - [close,1928] - {dataSource-1} closed\r",
          "host" => "DESKTOP-O93E7VQ",
    "@timestamp" => 2022-02-19T02:36:36.338Z,
          "path" => "E:/home/wxp/box/task/box-task-info.log"
}

通过kibana查看日志：​​​​​​http://localhost:5601/

可以从kibana看到如下日志信息：

4. 错误的配置文件

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
 
input {  
  file {
    start_position => end 
    path => "E:/home/wxp/box/task/box-task-info.log"
  } 
}
 
output {
  stdout{}
  elasticsearch {
   ​hosts => ["192.168.22.56:9200"]
    action => "index"
    index => "test_log_index"
  }
}

启动时，程序启动失败，出现类似如下的错误提示：

#启动配置
logstash -f log.conf


[2022-02-19T11:39:34,615][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2022-02-19T11:39:34,857][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", \"}\" at line 17, column 4 (byte 256) after output {\n  stdout{}\n  elasticsearch {\n   ", :backtrace=>["D:/sdks/logstash-7.15.1-windows-x86_64/logstash-7.15.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "D:/sdks/logstash-7.15.1-windows-x86_64/logstash-7.15.1/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "D:/sdks/logstash-7.15.1-windows-x86_64/logstash-7.15.1/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "D:/sdks/logstash-7.15.1-windows-x86_64/logstash-7.15.1/logstash-core/lib/logstash/agent.rb:391:in `block in converge_state'"]}
[2022-02-19T11:39:34,973][INFO ][logstash.runner          ] Logstash shut down.

根据提示，可以看出配置文件中出现了无效字符。

5. 验证配置文件正确性

logstash --config.test_and_exit --path.config log.conf

验证结果输出：

[2022-02-19T11:08:03,832][FATAL][logstash.runner          ] The given configuration is invalid. Reason: Expected one of [ \t\r\n], "#", [A-Za-z0-9_-], '"', "'", "}" at line 17, column 4 (byte 256) after output {
  stdout{}
  elasticsearch {

[2022-02-19T11:08:03,837][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
        at D_3a_.sdks.logstash_minus_7_dot_15_dot_1_minus_windows_minus_x86_64.logstash_minus_7_dot_15_dot_1.lib.bootstrap.environment.<main>(D:\sdks\logstash-7.15.1-windows-x86_64\logstash-7.15.1\lib\bootstrap\environment.rb:94) ~[?:?]

通过notepad++，重新打开文件，并显示所有符合的方式查看，发现对应位置出现了乱码字符。