kubernetes集群elasticsearch日志查询--java客户端开发

最新推荐文章于 2023-07-04 14:58:48 发布

xingyuzhe

最新推荐文章于 2023-07-04 14:58:48 发布

阅读量1.3k

点赞数

分类专栏： Kubernetes 文章标签： kubernetes elasticsearch java

本文链接：https://blog.csdn.net/xingyuzhe/article/details/81096362

版权

Kubernetes 专栏收录该内容

12 篇文章 4 订阅

订阅专栏

目标：基于elasticsearch api开发客户端，结合EFK日志框架，实现指定容器的日志收集

环境：IntelliJ IDEA

步骤：Elasticsearch概述->项目依赖pom.xml->elasticsearch api服务开发->运行测试

1.Elasticsearch概述

Elasticsearch是一个基于Lucene构建的开源、分布式、RESTful接口的全文搜索引擎。ES也是一个分布式文档数据库，其中每个字段均可被索引，每个字段的数据均可被搜索，可在短时间内存储、搜索和分析大量数据。

基本概念：

(1)集群 - cluster

集群是一个或多个节点的集合，其中节点共同保存整个数据，并在所有节点上提供联合索引和搜索功能。一个集群由一个唯一的群集ID确定。

(2)节点 - Node

节点是单个服务器实例，节点由一个UUID来标识

(3)索引 - Index

一个索引是一个具有相似特点的文档的集合。类似于关系型数据库中的数据库。

(4)类型 - type

一个索引内可以定义多个类型。一个类型就是一个索引的逻辑上的划分，通常定义一系列具有相同字段的文档结构。

(5)文档 - Document

文档是能够被检索的数据的最小单位，采用JSON的结构进行存储。

(6)分片与副本 - Shard&Replicas

每一个分片都是一个全功能的独立索引，可以被分布到集群中的任意节点。

2.项目依赖pom.xml

创建maven项目，修改pom.xml文件，新增api依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
 
    <groupId>com.boe.cloud</groupId>
    <artifactId>devclusterlog</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
 
    <name>devclusterlog</name>
    <description>search the pod log in k8s cluster</description>
 
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.3.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
 
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>
 
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
 
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
 
        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>6.2.4</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
        </dependency>
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger2</artifactId>
            <version>2.8.0</version>
        </dependency>
        <dependency>
            <groupId>io.springfox</groupId>
            <artifactId>springfox-swagger-ui</artifactId>
            <version>2.8.0</version>
        </dependency>
    </dependencies>
 
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

3.elasticsearch api服务开发

elasticsearch基于java编写，因此原生即支持java api

(1)ElasticsearchLogService,java

package com.boe.cloud.devclusterlog;
 
import org.elasticsearch.action.search.SearchRequestBuilder;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.index.query.BoolQueryBuilder;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.join.query.HasChildQueryBuilder;
import org.elasticsearch.join.query.JoinQueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.sort.SortOrder;
import org.elasticsearch.transport.client.PreBuiltTransportClient;
 
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.*;
 
public class ElasticsearchLogService {
 
    // 客户端连接
    private static TransportClient transportClient;
    // 索引库名
    private static String index = "";
    // 类型名称
    private static String type = "";
 
    // 初始化 - 连接elasticsearch数据库
    public String clientInit(String nodeIP, int port){
        String result = "Failed!";
        try {
            // setting对象指定集群配置信息
            Settings setting = Settings.builder()
                    .put("cluster.name", "kubernetes-logging")
                    .put("client.transport.sniff", false)
                    .build();
 
            // 创建客户端
            transportClient = new PreBuiltTransportClient(setting)
                    .addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(nodeIP),port));
 
            System.out.println("ElasticsearchClient 连接成功");
            result = "Success!";
        }catch (Exception e){
            e.printStackTrace();
        }
        return result;
    }
 
    // 列出节点信息
    public List<String> getNodeinfo(){
        List<String> nodeInfo = new ArrayList<>();
        try {
            List<DiscoveryNode> connectedNodes = transportClient.connectedNodes();
            for (DiscoveryNode discoveryNode : connectedNodes){
                nodeInfo.add(discoveryNode.getHostName());
                System.out.println(discoveryNode.getHostName());
            }
        }catch (Exception e){
            e.printStackTrace();
        }
        return nodeInfo;
    }
 
    // Map按Key进行排序
    public static Map<String, String> sortMapByKey(Map<String, String> map){
        if (map == null || map.isEmpty()) {
            return null;
        }
 
        Map<String, String> sortMap = new TreeMap<String, String>(
                new MapKeyComparator());
 
        sortMap.putAll(map);
 
        return sortMap;
    }
 
    // 获取表信息
    public List<String> searchIndexinfo(String id, String tp, String nsName, String podName){
 
        // 指定要查询的index和type
        SearchRequestBuilder search = transportClient.prepareSearch(id).setTypes(tp);
        // 使用原生排序优化性能
        search.addSort("_doc", SortOrder.ASC);
        // 设置每批读取的数据量
        search.setSize(5);
        // 默认查询所有
        //search.setQuery(QueryBuilders.queryStringQuery("*:*"));
        //search.setQuery(QueryBuilders.termQuery("_id", "IPdwkWQBjHvJN4hY1OSQ"));
        BoolQueryBuilder bqb = QueryBuilders.boolQuery();
        bqb.must(QueryBuilders.matchQuery("kubernetes.namespace_name", nsName));
        bqb.must(QueryBuilders.matchQuery("kubernetes.pod_name", podName));
        //search.setQuery(QueryBuilders.boolQuery().must(QueryBuilders.matchQuery("kubernetes.pod_name", "elasticsearch-logging-1")));
        search.setQuery(bqb);
        // 设置维护1分钟有效期
        search.setScroll(TimeValue.timeValueMinutes(1));
 
        // 获取首次查询结果
        SearchResponse scrollResp = search.get();
        //打印命中数量
        System.out.println("命中总数量："+scrollResp.getHits().getTotalHits());
        //打印计数
        int count=1;
        List<Map<String, Object>> sourceList = new ArrayList<>();
        Map<String, String> logSort = new HashMap<>();
        //List<String> logList = new ArrayList<>();
        do {
            System.out.println("第"+count+"次打印数据：");
            //读取结果集数据
            for (SearchHit hit : scrollResp.getHits().getHits()) {
                Map<String, Object> source = hit.getSource();
                sourceList.add(source);
                //logList.add(source.get("log").toString());
                logSort.put(source.get("@timestamp").toString(), source.get("log").toString());
                System.out.println(source);
            }
            count++;
            //将scorllId循环传递
            scrollResp = transportClient.prepareSearchScroll(scrollResp.getScrollId()).setScroll(TimeValue.timeValueMinutes(1)).execute().actionGet();
 
            //当searchHits的数组为空的时候结束循环，至此数据全部读取完毕
        } while(scrollResp.getHits().getHits().length != 0);
        //return sourceList;
        //Collections.reverse(logList);
        Map<String, String> resultMap = sortMapByKey(logSort);
 
        // 获取value list
        List<String> valueList = new ArrayList<String>();
        Iterator<String> it = resultMap.keySet().iterator();
        while (it.hasNext()) {
            String key = it.next().toString();
            valueList.add(resultMap.get(key));
        }
        Collections.reverse(valueList);
 
        //return logList;
        //return resultMap;
        return valueList;
    }
 
}

(2)ElasticsearchLogController.java

package com.boe.cloud.devclusterlog;
 
import io.swagger.annotations.ApiOperation;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
 
import java.net.UnknownHostException;
import java.util.List;
import java.util.Map;
 
@RestController
@RequestMapping("/test")
public class ElasticsearchLogController {
 
    ElasticsearchLogService service = new ElasticsearchLogService();
 
    @ApiOperation(value = "Connect", notes = "Connect")
    @RequestMapping(value = "/connect", method = RequestMethod.GET)
    public String init(@RequestParam(value = "NodeIP") String ip,
                       @RequestParam(value = "Port") int port){
        return service.clientInit(ip, port);
    }
 
    @ApiOperation(value = "NodeIP", notes = "NodeIP")
    @RequestMapping(value = "/getnode", method = RequestMethod.GET)
    public List<String> getNode(){
        return service.getNodeinfo();
    }
 
    @ApiOperation(value = "SearchLog", notes = "SearchLog")
    @RequestMapping(value = "/searchlog", method = RequestMethod.POST)
    public List<String> searchInfo(@RequestParam(value = "Index") String index,
                                          @RequestParam(value = "Type") String type,
                                          @RequestParam(value = "NamespaceName") String nsName,
                                          @RequestParam(value = "PodName") String podName){
        return service.searchIndexinfo(index, type, nsName, podName);
    }
}

(3)MapKeyComparator.java

按时间排序

package com.boe.cloud.devclusterlog;
 
import java.util.Comparator;
 
public class MapKeyComparator implements Comparator<String> {
    @Override
    public int compare(String str1, String str2) {
 
        return str1.compareTo(str2);
    }
 
}

4.运行测试

传入参数：

获取已排序日志：

以上，测试完成。

xingyuzhe

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
2
评论
kubernetes集群elasticsearch日志查询--java客户端开发

目标：基于elasticsearch api开发客户端，结合EFK日志框架，实现指定容器的日志收集环境：IntelliJ IDEA步骤：Elasticsearch概述-&gt;项目依赖pom.xml-&gt;elasticsearch api服务开发-&gt;运行测试1.Elasticsearch概述Elasticsearch是一个基于Lucene构建的开源、分布式、RESTful...
复制链接

扫一扫