2022-ES学习与实践

ES学习系列
核心场景：es5.x基于Flink-Connector-Elasticsearch实现EsSink
问题：添加权限控制，即x-pack相关实现。
解决：
ES5.x中，与es交互的client是TransportClient。
以下为flink-connector-es5源码：

    @Override
    public TransportClient createClient(Map<String, String> clientConfig) {
        Settings settings =
                Settings.builder()
                        .put(NetworkModule.HTTP_TYPE_KEY, Netty3Plugin.NETTY_HTTP_TRANSPORT_NAME)
                        .put(NetworkModule.TRANSPORT_TYPE_KEY, Netty3Plugin.NETTY_TRANSPORT_NAME)
                        .put(clientConfig)
                        .build();

        TransportClient transportClient = new PreBuiltTransportClient(settings);
        for (TransportAddress transport :
                ElasticsearchUtils.convertInetSocketAddresses(transportAddresses)) {
            transportClient.addTransportAddress(transport);
        }

        return transportClient;
    }

PreBuiltTransportClient客户端中不包含x-pack相关实现，若要实现需要额外添加部分依赖

    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>5.5.3</version>
    </dependency>
    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>x-pack-transport</artifactId>
      <version>5.5.3</version>
    </dependency>

重写Elasticsearch5ApiCallBridge类中的createClient方法：

    @Override
    public TransportClient createClient(Map<String, String> clientConfig) {
        Settings settings =
                Settings.builder()
                        .put(CONFIG_CLUSTER_NAME, this.clusterName)
                        .put(CONFIG_KEY_TRANSPORT_SSL, false)
                        .put(CONFIG_KEY_TRANSPORT_SNIFF, true)
                        // .put(NetworkModule.HTTP_TYPE_KEY, Netty3Plugin.NETTY_HTTP_TRANSPORT_NAME)
                        // .put(NetworkModule.TRANSPORT_TYPE_KEY, Netty3Plugin.NETTY_TRANSPORT_NAME)
                        .put(clientConfig)
                        .build();
        // PreBuiltXPackTransportClient需要导入x-pack包
        TransportClient transportClient = new PreBuiltXPackTransportClient(settings);
        for (TransportAddress transport : ElasticsearchUtils.convertInetSocketAddresses(transportAddresses)) {
            transportClient.addTransportAddress(transport);
        }
        return transportClient;
    }

小结：
es5的客户端为TransportClient而不是RestHighLevelClient，TransportClient需要指定集群名称，连接端口需要是transport.tcp.port而不是http.port，这两个问题都会导致集群连接失败。
解决es5权限认证相关问题分两步：

1. 引入x-pack相关依赖
2. 重写createClient方法，把transportClient的实现类改为PreBuiltXPackTransportClient。

存疑：Flink-Connector-Elasticsearch5源码中Elasticsearch5ApiCallBridge类中的createClient方法中下面这两个连接参数不知道有啥作用

.put(NetworkModule.HTTP_TYPE_KEY, Netty3Plugin.NETTY_HTTP_TRANSPORT_NAME)
.put(NetworkModule.TRANSPORT_TYPE_KEY, Netty3Plugin.NETTY_TRANSPORT_NAME)