公司软件平台对接华为FI集群,使用spark集群是报错,在spark客户端上执行提交命令报错如下
./Spark2x/spark/bin/spark-submit -master yarn --class org.apache.spark.examples.SparkPi --master yarn-client ./Spark2x/spark/examples/jars/spark-examples_2.12-3.1.1-h0.cbu.mrs.313.r10.jar
数据处理平台日志显示
| ERROR | main | Failed to get response, Error is :
javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to
find valid certification path to requested target |
org.apache.ranger.admin.client.RangerAdminRESTClient.run(RangerAdminRESTClient.java:234
经过排查发现,需要修改spark客户端配置文件,问题处理完成
Spark2x/spark/conf/spark-default.conf
spark.ranger.plugin.authorization.enable=false