dataFrame（DF）将数据插入ES时报错 org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect E

最新推荐文章于 2024-01-23 18:58:00 发布
最新推荐文章于 2024-01-23 18:58:00 发布
阅读量1.8k 收藏 3
点赞数 1
分类专栏： Scala SparkSQL 文章标签： hadoop elasticsearch spark scala
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_45441740/article/details/127867694
版权

SparkSql Elasticsearch 权限错误 解决方法 配置参数
关键词由CSDN通过智能技术生成

SparkSql插入ES时报错解决（针对增加用户权限之后的报错解决）

org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting ‘es.nodes.wan.only’

以下是报错信息

Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:340)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:97)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.spark.sql.ElasticsearchRelation.insert(DefaultSource.scala:620)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.spark.sql.DefaultSource.createRelation(DefaultSource.scala:107)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:86)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:131)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:127)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:155)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:152)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:127)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.DataFrameWriter$$anonfun$runCommand$1.apply(DataFrameWriter.scala:676)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SQLExecution$$anonfun$withNewExecutionId$1.apply(SQLExecution.scala:78)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:125)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:73)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:676)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:285)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:271)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:229)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at com.pukka.bigdata.dealwith.toes.SparkToEs$.main(SparkToEs.scala:112)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at com.pukka.bigdata.dealwith.toes.SparkToEs.main(SparkToEs.scala)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15-11-2022 14:43:56 CST playrecordtoes INFO - Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: org.elasticsearch.hadoop.rest.EsHadoopRemoteException: security_exception: missing authentication credentials for REST request [/]
15-11-2022 14:43:56 CST playrecordtoes INFO - null
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.RestClient.checkResponse(RestClient.java:469)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:426)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:735)
15-11-2022 14:43:56 CST playrecordtoes INFO - 	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:330)

报错解决
在配置参数中加入 “es.net.http.auth.user”、 “es.net.http.auth.pass” 即可

val options = Map(
      //如果没有index,则新建
      "es.index.auto.create" -> "true",
      "es.nodes.wan.only" -> "true",
      "es.nodes" -> "192.168.20.131,192.168.20.130",
      //"es.nodes" -> "10.0.6.56",
      //"es.nodes" -> "60.167.225.130",
      //es的端口
      "es.port" -> "9200",
      "es.net.http.auth.user" ->"elastic",
      "es.net.http.auth.pass" ->"test@2022!"
    )
    val esPath=s"play_record_$year$month/_doc"
    dataFrame
      .write
      .format("org.elasticsearch.spark.sql")
      .options(options)
      .mode(SaveMode.Append)
      //index为epg_itv_play_record_$year$month
      .save(esPath)
    spark.stop()
墨兰公子
关注
  • 1
    点赞
  • 踩
  • 3
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
专栏目录
es在hive建外表报错：Cannot detect ES version
weixin_39419040的博客
08-28 1988
在hive建es外表时遇到了一个错误： 在建表语句中增加参数'es.nodes.wan.only' = 'true'之后还是报同样的错误。 可是按理说之前在相同集群建es外表时候没有出现这样的错误。 进入hive 的debug模式： hive -hiveconf hive.root.logger=DEBUG,console 可以看到日志输出：Unsupported/Unknown Elasticsearch version 6.8.2 想到可能是因为elasticsearch-hado..
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: expected end of object, found END_ARRAY
zhaijiajia94的博客
08-30 3056
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: expected end of object, found END_ARRAY at org.elasticsearch.hadoop.util.Assert.isTrue(Assert.java:60) at org.elasticsearc...
Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not acces
ZMC921的博客
06-17 5763
hive 给es写数据，建表的时候报下面的错误 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster i
elasticsearch6升级7后hive映射表报错es hadoop illegal argument canot detect es version 问题解决
xiaohai798的专栏
08-08 956
elasticsearch6升级7后hive映射表报错es hadoop illegal argument canot detect es version 问题解决
ES插入数据报错 [2019-08-28T16:52:00,257][INFO ][logstash.outputs.elasticsearch] retrying failed action wit
HouYing
08-28 882
https://github.com/NLPchina/elasticsearch-sql [2019-08-28T16:52:00,257][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 403 ({"type"=>"cluster_block_exception", "...
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException
DPnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnice的博客
05-10 6440
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot determine write shards for [index/type]; likely its format is incorrect (maybe it contains illegal characters?)
elasticsearch-hadoop-6.8.23.zip
05-08
6. **数据迁移和同步**：Elasticsearch-Hadoop支持将HDFS上的数据批量导入到Elasticsearch，或者将Elasticsearch中的数据导出到HDFS，方便进行数据备份、迁移和同步。 7. **容错性和扩展性**：由于Elasticsearch和...
pyspark.sql.DataFrame与pandas.DataFrame之间的相互转换实例
01-21
代码如下，步骤流程在代码注释中可见： # -*- coding: utf-8 -*- import pandas as pd ...df = pd.DataFrame([[1, 2, 3], [4, 5, 6]], index=['row1', 'row2'], columns=['c1', 'c2', 'c3']) #打印数据 pri
pandas数据选取：df[] df.loc[] df.iloc[] df.ix[] df.at[] df.iat[]
09-17
本文将详细介绍DataFrame中用于数据选取的六种方法：`df[]`、`df.loc[]`、`df.iloc[]`、`df.ix[]`、`df.at[]`和`df.iat[]`。 1. **行（列）选取：df[]** `df[]` 用于单一维度上的数据选取，即选取行或列。...
将pandas.dataframe的数据写入到文件中的方法
12-25
【import pandas as pd】pandas用来做数据处理。【import numpy as np】numpy用来做高维度矩阵运算.【import matplotlib.pyplot as plt】matplotlib用来做数据可视化。 pandas数据写入到csv文件中： 【names = [...
detect-es-version
03-21
detect-es-version 给定软件包名称，文件路径或代码字符串，则返回。您可以试用网络应用程序版本！ 用法 const { getEcmaVersion } = require ( 'detect-es-version' ) ; const ecmaVersion = getEcmaVersion ( 'const foo = 123;' ) ; console . log ( ecmaVersion ) ; // prints 2015 原料药 getPackageEcmaVersion（packageString，resolverOptions） packageString （ String ）：表示包装的字符串。例如，要从npm中提取“ react”，“ react@16.13.1”。 resolverOptions （ Object ）：一个选项对象，传递给。这可用
数据挖掘技术与应用：查看DataFrame数据的常用属性.pdf
02-20
数据挖掘技术与应用：查看DataFrame数据的常用属性 数据挖掘技术是指从大量数据中自动发现有价值的知识或模式的过程。Pandas 库是 Python 中最常用的数据处理工具之一，它提供了多种数据结构和操作方法。本文档主要...
ES 数据插入异常原因分析
最新发布
J 博客
01-23 1508
Springboot 执行 ES 单元测试时发生上述异常 , 看提示应该是因为使用了 ignore_throttled 过期参数导致的，但是我的查询并没有使用这个字段，那它是从哪里来的呢？先 debug 查看异常处代码 Response.assertWarningValue 这段代码是从 es 请求的响应中提取告警信息，判断是否符合断言，该正则表达式分为三部分对比运行时异常信息发现，299 Elasticsearch-7.17.13-2b211dbb8bfdecaf7f5b44d356bdfe54b10
spring调用es插入报错以及解决办法
w876484067的博客
10-14 1310
插入是报错如下 org.elasticsearch.discovery.MasterNotDiscoveredException: null 解决办法修改es配置文件 如下，改完之后成功插入，有遇到的这个问题的可以试试
SpringBoot操作ElasticSsearch添加数据时报错
搬砖爱好者.的博客
11-08 675
测试es添加数据方法 public void test() throws IOException { // 创建索引 CreateIndexRequest request = new CreateIndexRequest("test"); client.indices().create(request, RequestOptions.DEFAULT); // 创建对象 IndexVo indexVo = new IndexVo("666"); // test为上面创建的
Spark操作Es报错：sHadoopIllegalArgumentException: resource must be specified for Elasticsearch resources
笑里笑外~
12-04 1023
项目场景： 使用spark sql 操作读取 es中的数据 版本： es ：6.4.2 es-hadoop: 6.4.2 spark:2.1.1 部分代码： Map<String,String> cfg = new HashMap(); cfg.put("es.query","...."); cfg.put("es.read.field.incloude","...") Dataset ds = sparkSession.read().format("org.elasticsearch.spar
ElasticSearch集群升级Spark出现EsHadoopIllegalArgumentException:Unsupported/Unknown Elasticsearch version解决
CircleLee的博客
09-02 1786
近期由于平台提供方ES集群升级，由5.5.0升级为6.4.3 原SparkStreaming访问ES使用elasticsearch-hadoop依赖中EsSpark.saveJsonToEs写入数据。 将elasticsearch-hadoop依赖包升级为对应6.x版本后esMap添加Xpack参数: es.nodes es.port es.mapping.id es.net.http.auth.user es.net.http.auth.pass 本地调试读写均正常，但在提交yarn集群运
JavaEsSpark.esJsonRDD函数读取ES数据报错
liulu930507的博客
11-17 6634
错误信息如下Exception in thread "main" org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or
安装es(elasticsearch)遇到问题总结
徐海兴的专栏
05-25 786
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536] [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] [3]...
pandas，将df.iloc[1:2,1:2]转化为e-12科学计数法
06-10
如果您想将 DataFrame 中的某个区域转换为科学计数法的 `e-12` 表示，可以使用以下代码： ```python import pandas as pd import numpy as np # 假设您的 DataFrame 名称为 df df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6], 'C': [7.123456789e-12, 8.123456789e-12, 9.123456789e-12]}) # 指定区域为 [1:2, 1:2]，即第 2 行、第 2 列的单个单元格 df.iloc[1:2, 1:2] = '{:.12e}'.format(df.iloc[1:2, 1:2].values[0][0]) print(df) ``` 输出结果为： ``` A B C 0 1 4 7.123456e-12 1 2 5 8.000000e-12 2 3 6 9.123457e-12 ``` 这样就可以将 DataFrame 中指定区域内的每一个数值转换为 `e-12` 表示。其中，`12` 表示保留小数点后 12 位有效数字，您可以根据需要调整该数字。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值