spark与elasticsearch整合netty冲突

spark与elasticsearch整合netty冲突

报错信息：

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
Exception in thread "elasticsearch[_client_][generic][T#3]" java.lang.NoSuchMethodError: io.netty.bootstrap.Bootstrap.config()Lio/netty/bootstrap/BootstrapConfig;
	at org.elasticsearch.transport.netty4.Netty4Transport.lambda$stopInternal$7(Netty4Transport.java:473)
	at org.apache.lucene.util.IOUtils.close(IOUtils.java:89)
	at org.elasticsearch.common.lease.Releasables.close(Releasables.java:36)
	at org.elasticsearch.common.lease.Releasables.close(Releasables.java:46)
	at org.elasticsearch.common.lease.Releasables.close(Releasables.java:51)
	at org.elasticsearch.transport.netty4.Netty4Transport.stopInternal(Netty4Transport.java:456)
	at org.elasticsearch.transport.TcpTransport.lambda$doStop$5(TcpTransport.java:956)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:575)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
19/09/23 21:28:43 ERROR ApplicationMaster: User class threw exception: java.lang.IllegalStateException: thread was not started
java.lang.IllegalStateException: thread was not started
	at io.netty.util.concurrent.GlobalEventExecutor.awaitInactivity(GlobalEventExecutor.java:201)
	at org.elasticsearch.transport.client.PreBuiltTransportClient.close(PreBuiltTransportClient.java:142)
	at RunMain.main(RunMain.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:646)
19/09/23 21:28:43 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.IllegalStateException: thread was not started)
19/09/23 21:28:43 INFO SparkContext: Invoking stop() from shutdown hook

添加获取client的setting参数
.put(“transport.type”, “netty3”)//解决与SPARK中的netty版本不一致的问题
.put(“http.type”, “netty3”)
.put(“cluster.name”, “xxxxx”)//集群模式需要添加此参数

获取客户端如下：

	 /**
     * 获取client
     */
    public static TransportClient getClusterClient(String clusterName, String []esIps) throws UnknownHostException {
        Settings settings = Settings.builder()
                .put("cluster.name", clusterName)
                .put("client.transport.ping_timeout", "60s")
                .put("transport.type", "netty3")//解决与SPARK中的netty版本不一致的问题
                .put("http.type", "netty3")
                .put("cluster.name", "xxxxx")//集群模式需要添加此参数
                .build();
        PreBuiltTransportClient client = new PreBuiltTransportClient(settings);

        // round robin 轮流使用每个地址
        for (String esIp : esIps) {
            client.addTransportAddress(
                    new InetSocketTransportAddress(InetAddress.getByName(esIp.split(":")[0]), Integer.parseInt(esIp.split(":")[1])));
        }

        return client;
    }

注意：xxxxx 要修改为对应es集群的 cluster.name