springboot+elk

ELK搭建

ELK主要包括三大部分：elasticsearch、logstash、kibana，其中elasticsearch和kibana已经搭建完成，详情Elasticsearch 6.2.2安装部署+kibana+HQ，现需要搭建logstash：

下载logstash

https://www.elastic.co/cn/downloads/logstash

logstash配置与启动

logstash版本应该与elasticsearch一致，本文选择6.2.2，下载后解压，进入config文件夹，新建配置文件：

cd logstash-6.2.2/config
vim log4j_to_es.conf

input {
  tcp {
    mode => "server"
    host => "0.0.0.0"      #允许不同来源访问
    type => "test1"        #区分不同服务
    port => 4560		   #不同服务访问端口不同
    codec => json_lines
  }
  tcp {
    mode => "server"
    host => "0.0.0.0"
    type => "test2"
    port => 4561
    codec => json_lines
  }
}
filter {
  #Only matched data are send to output.
}
output {
  if [type] == "test1" {         # 区分微服务
    elasticsearch {
      action => "index"          # 输出时创建映射
      hosts  => "localhost:9200"   # ElasticSearch 的地址和端口
      index  => "applog"         # 指定索引名
      codec  => "json"
     }
  }
  if [type] == "test2" {
    elasticsearch {
      action => "index"          
      hosts  => "localhost:9200"   
      index  => "applog2"         
      codec  => "json"
     }
  }
}

配置完成后，采用此配置文件启动，出现如下异常：

./bin/logstash -f /home/yzh/logstash-6.2.2/config/log4j_to_es.conf


[2021-03-15T17:35:57,879][ERROR][logstash.plugins.registry] Tried to load a plugin's code, but failed. {:exception=>#<LoadError: no such file to load -- logstash/inputs/log4j>, :path=>"logstash/inputs/log4j", :type=>"input", :name=>"log4j"}
[2021-03-15T17:35:57,902][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::PluginLoadingError", :message=>"Couldn't find any input plugin named 'log4j'. Are you sure this is correct? Trying to load the log4j input plugin resulted in this error: no such file to load -- logstash/inputs/log4j", :backtrace=>["/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/plugins/registry.rb:192:in `lookup_pipeline_plugin'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/plugin.rb:140:in `lookup'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/plugins/plugin_factory.rb:81:in `plugin'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:112:in `plugin'", "(eval):8:in `<eval>'", "org/jruby/RubyKernel.java:994:in `eval'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:84:in `initialize'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/home/yzh/logstash-6.2.2/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/home/yzh/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

经查询，是由于缺少log4j插件造成的，因此需要安装插件，首先配置Gemfile文件，修改source为国内镜像，之后利用logstash-plugin安装插件：

vim Gemfile
source "https://mirrors.tuna.tsinghua.edu.cn/rubygems/"

./bin/logstash-plugin install logstash-input-log4j

安装完成后即可成功启动

springboot配置

添加依赖

在pom.xml中添加如下内容：

    	<properties>
        	<ch.qos.logback.version>1.2.3</ch.qos.logback.version>
    	</properties>		
		<!--   日志采集配置     -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-logging</artifactId>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-core</artifactId>
            <version>${ch.qos.logback.version}</version>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>${ch.qos.logback.version}</version>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-access</artifactId>
            <version>${ch.qos.logback.version}</version>
        </dependency>
        <dependency>
            <groupId>net.logstash.logback</groupId>
            <artifactId>logstash-logback-encoder</artifactId>
            <version>5.1</version>
        </dependency>

增加日志配置

在logback.xml中增加如下配置：

    <appender name="stash" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>10.0.0.0:4560</destination>
        <!-- encoder必须配置,有多种可选 -->
        <encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder" />
    </appender>

<!-- 在不同级别的日志下面添加输出选项 -->
    <!--开发/测试环境下-->
    <springProfile name="test, dev">
        <logger name="com.qwe.smart" level="DEBUG" additivity="false">
            <appender-ref ref="consoleLog" />
            <appender-ref ref="infoLog"/>
            <appender-ref ref="errorLog"/>
            <appender-ref ref="stash"/>
        </logger>
    </springProfile>

    <!-- 生产环境. -->
    <springProfile name="prod">
        <logger name="com.qwe.smart" level="INFO" additivity="false">
            <appender-ref ref="consoleLog" />
            <appender-ref ref="infoLog"/>
            <appender-ref ref="errorLog"/>
            <appender-ref ref="stash"/>
        </logger>
    </springProfile>
    

日志输出测试

在代码中增加日志输出

        LOGGER.info("测试日志收集1");
        LOGGER.info("测试日志收集2");
        LOGGER.info("测试日志收集3");
        LOGGER.info("测试日志收集4");
        LOGGER.info("测试日志收集5");
        

通过kibana检索和查看日志