SpringBoot项目,从filebeat中直接导日志数据到elasticsearch中,查询获取数据

@hpx

SpringBoot项目,从filebeat中直接导入日志数据到elasticsearch中,并查询获取数据

前提,笔者使用的是elasticsearch是6.2.4版本的,所以filebeat也是6.2.4版本的;
如果使用其他版本,有可能会报错;

Exiting: Couldn’t connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://blahblahblha: Connection marked as failed because the onConnect callback failed: This Beat requires the default distribution of Elasticsearch. Please upgrade to the default distribution of Elasticsearch from elastic.co, or downgrade to the oss-only distribution of beats

安装Filebeat

1 .linux方式:

下载:curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

2.解压到usr/local目录下

tar zxvf filebeat-7.3.1-linux-x86_64.tar.gz -C /usr/local

3.导入模拟错误日志

/users/liuxg/alarm.log.2019-11-07

4.使用notepad等工具,复制filebeat.yml文件成filebeat_es.yml,并进行修改

ilebeat.inputs:
- type: log
enabled: true
paths:
- /users/liuxg/alarm.log.2019-11-07

这里的ip设置成你虚拟机的端口号:

output.elasticsearch:
hosts: [“192.168.25.131:9200”]

5.这里的配置非常简单。在上面的paths里，它指向我们的存放的log文件。当你们自己做这个练习的时候，请修改这个文件的路径。我们可以直接运行filebeat：
先启动elasticsearch

[es@jasper bin]$ ./elasticsearch

elasticsearch启动成功显示如下:

$ ./filebeat -c filebeat_es.yml

5.1如果遇到error可以通过以下命令检查filebeat是否正常(在filebeat目录下)

./filebeat -c filebeat_es.yml test output

结果如下:

elasticsearch: http://192.168.25.131:9200…
parse url… OK
connection…
parse host… OK
dns lookup… OK
addresses: 192.168.25.131
dial up… OK
TLS… WARN secure connection disabled
talk to server… OK
version: 6.2.4

6.在elasticsearch-head中可获得查询到以下结果

{
“_index”: “filebeat-6.2.4-2019.11.27”,
“_type”: “doc”,
“_id”: “-2qAq24B-pTOLfkoe_AR”,
“_version”: 1,
“_score”: 1,
“_source”: {
“@timestamp”: “2019-11-27T06:15:52.833Z”,
“prospector”: {
“type”: “log”
},
“beat”: {
“name”: “jasper”,
“hostname”: “jasper”,
“version”: “6.2.4”
},
“source”: “/users/liuxg/alarm.log.2019-11-07”,
“offset”: 1172,
“message”: "******************************************************
***********************************"
}
}

7.idea环境搭建

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>elk_kafka</artifactId>
        <groupId>com.elk</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>kafka_search</artifactId>
    <dependencies>
        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-elasticsearch</artifactId>
        </dependency>
    </dependencies>

</project>

8.配置文件

server:
  port: 9302
spring:
  application:
    name: kafka-search
  data:
    elasticsearch:
      cluster-nodes: 192.168.25.131:9300

9.启动类实现

package com.elk.search;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SearchApplication {
    public static void main(String[] args) {
        SpringApplication.run(SearchApplication.class);
    }
}

10.测试pojo类实现

import java.io.Serializable;
import java.sql.Timestamp;

@Data
@AllArgsConstructor
@NoArgsConstructor
@Document(indexName = "filebeat-6.2.4-2019.11.27",type = "doc")
public class RuningWater implements Serializable {

    @Id
    private Long id;

    //日志报文的打印时间
    @Field(index = true)
    private Timestamp timestamp;

    //日志信息
    @Field(index = true)
    private String message;
}

11.Dao层实现

package com.elk.search.dao;

import com.elk.search.pojo.RuningWater;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;

import java.util.List;

public interface RunWaterSearchDao extends ElasticsearchRepository<RuningWater,String> {

    //根据方法名自动生成查询语句
    //http://localhost:9200/runwater/search/_search?message=xxx

     List<RuningWater> findByMessageLike(String message);

    
}

12.service层实现

import java.util.List;
@Service
public class RunWaterSearchService {

    @Autowired
    private RunWaterSearchDao runWaterSearchDao;

    //查询流水记录
    public List<RuningWater> getRuningWaterFrom1(String message){

        return runWaterSearchDao.findByMessageLike(message);

13.controller层实现

import com.elk.search.pojo.RuningWater;
import com.elk.search.service.RunWaterSearchService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@RestController
@CrossOrigin
@RequestMapping("/runwater")
public class RunWaterSearchController {

    @Autowired
    private RunWaterSearchService runWaterSearchService;

    //对流水日志进行搜索
    @RequestMapping(value = "/search/{message}",method = RequestMethod.GET)
    public List<RuningWater> search(@PathVariable String message){
        System.out.println(message);
        List<RuningWater> runingWaterFrom1 = runWaterSearchService.getRuningWaterFrom1(message);
        return runingWaterFrom1;
    }
}