spark连接es时。我尝试使用elasticsearch-spark_2.11去连接es。
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark_2.11</artifactId>
<version>2.1.0</version>
</dependency>
但时一直提示
Cannot find node with id [0UzWcOlOSh-WGCLVsTpAWA] (is HTTP enabled) from shard
之类的错误。 大体的意思是没有连接到es的节点 , 是否设置开启了es的http?
经过各种版本的尝试,原来是因为 spark与es的连接包的版本问题。版本更新成elasticsearch-spark-20_2.11 问题解决
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_2.11</artifactId>
<version>5.1.2</version>
</dependency>
spark连接代码:
SparkSession spark = SparkSession.builder().master("local[*]").appName("SparkCalculateRecommend").
config("spark.mongodb.input.uri", "mongodb://xm:xm123@192.168.31.203:27018/sns.igomoMemberInfo_Spark_input")
.config("spark.mongodb.output.uri", "mongodb://xm:xm123@192.168.31.203:27018/sns.igomoMemberInfo_Spark_output")
.config("spark.executor.memory", "512mb")
.config("es.nodes", "192.168.53.92")
.config("es.port", "9200")
.config("es.nodes.wan.only","true")
.getOrCreate();
JavaSparkContext jsc = new JavaSparkContext(spark.sparkContext());
EsSpark.saveToEs(gradeResultJavaRDD.rdd(),"sparkresult/sparkresult");
jsc.close();