端口多次重试失败异常

最新推荐文章于 2022-05-07 20:21:03 发布
最新推荐文章于 2022-05-07 20:21:03 发布
阅读量1.1k 收藏 2
点赞数 1
文章标签： spark
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/u013727054/article/details/118213565
版权 
11-08-2020 11:38:48 CST t_uac_user INFO - Starting job t_uac_user at 1597117128685
11-08-2020 11:38:48 CST t_uac_user INFO - job JVM args: '-Dazkaban.flowid=order' '-Dazkaban.execid=130' '-Dazkaban.jobid=t_uac_user'
11-08-2020 11:38:48 CST t_uac_user INFO - user.to.proxy property was not set, defaulting to submit user azkaban
11-08-2020 11:38:48 CST t_uac_user INFO - Building command job executor. 
11-08-2020 11:38:48 CST t_uac_user INFO - Memory granted for job t_uac_user
11-08-2020 11:38:48 CST t_uac_user INFO - 1 commands to execute.
11-08-2020 11:38:48 CST t_uac_user INFO - cwd=/opt/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/executions/130
11-08-2020 11:38:48 CST t_uac_user INFO - effective user is: azkaban
11-08-2020 11:38:48 CST t_uac_user INFO - Command: ssh mz-hadoop-01 "source /etc/profile;/mnt/db_file/wfs/paascloud/t_uac_user.sh  "
11-08-2020 11:38:48 CST t_uac_user INFO - Environment variables: {JOB_OUTPUT_PROP_FILE=/opt/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/executions/130/t_uac_user_output_8692020296885557590_tmp, JOB_PROP_FILE=/opt/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/executions/130/t_uac_user_props_3322224465573219858_tmp, KRB5CCNAME=/tmp/krb5cc__order__order__t_uac_user__130__azkaban, JOB_NAME=t_uac_user}
11-08-2020 11:38:48 CST t_uac_user INFO - Working directory: /opt/azkaban/azkaban-3.81.0/azkaban-exec-server/build/install/azkaban-exec-server/executions/130
11-08-2020 11:38:48 CST t_uac_user INFO - Spawned process with id 22057
11-08-2020 11:38:48 CST t_uac_user INFO - 起始日期:2020-08-10,结束日期:2020-08-10
11-08-2020 11:38:51 CST t_uac_user INFO - 20/08/11 11:38:51 INFO spark.SparkContext: Running Spark version 2.4.0.cloudera2
11-08-2020 11:38:51 CST t_uac_user INFO - 20/08/11 11:38:51 INFO spark.SparkContext: Submitted application: t_uac_user
11-08-2020 11:38:51 CST t_uac_user INFO - 20/08/11 11:38:51 INFO spark.SecurityManager: Changing view acls to: root
11-08-2020 11:38:51 CST t_uac_user INFO - 20/08/11 11:38:51 INFO spark.SecurityManager: Changing modify acls to: root
11-08-2020 11:38:51 CST t_uac_user INFO - 20/08/11 11:38:51 INFO spark.SecurityManager: Changing view acls groups to: 
11-08-2020 11:38:51 CST t_uac_user INFO - 20/08/11 11:38:51 INFO spark.SecurityManager: Changing modify acls groups to: 
11-08-2020 11:38:51 CST t_uac_user INFO - 20/08/11 11:38:51 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.Utils: max retries is 16
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.Utils: Successfully started service 'sparkDriver' on port 37149.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO spark.SparkEnv: Registering MapOutputTracker
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO spark.SparkEnv: Registering BlockManagerMaster
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-031bffb6-fbb3-4aaa-bfb3-f6213e01d7f4
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO spark.SparkEnv: Registering OutputCommitCoordinator
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.log: Logging initialized @3278ms
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO server.Server: jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO server.Server: Started @3449ms
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.Utils: max retries is 16
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4046. Attempting port 4047.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4047. Attempting port 4048.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4048. Attempting port 4049.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4049. Attempting port 4050.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4050. Attempting port 4051.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4051. Attempting port 4052.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4052. Attempting port 4053.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4053. Attempting port 4054.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4054. Attempting port 4055.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 WARN util.Utils: Service 'SparkUI' could not bind on port 4055. Attempting port 4056.
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 ERROR ui.SparkUI: Failed to bind SparkUI
11-08-2020 11:38:52 CST t_uac_user INFO - java.net.BindException: Address already in use: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.nio.ch.Net.bind0(Native Method)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.nio.ch.Net.bind(Net.java:433)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.nio.ch.Net.bind(Net.java:425)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.spark_project.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:351)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.spark_project.jetty.server.ServerConnector.open(ServerConnector.java:319)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.spark_project.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.spark_project.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$newConnector$1(JettyUtils.scala:353)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$httpConnect$1(JettyUtils.scala:382)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:385)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.ui.JettyUtils$$anonfun$7.apply(JettyUtils.scala:385)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:2315)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2307)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:385)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.ui.WebUI.bind(WebUI.scala:132)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:452)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.SparkContext$$anonfun$11.apply(SparkContext.scala:452)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at scala.Option.foreach(Option.scala:257)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.SparkContext.<init>(SparkContext.scala:452)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at scala.Option.getOrElse(Option.scala:121)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at com.mingzhi.universal.load_all_from_mysql$.load(load_all_from_mysql.scala:73)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at com.mingzhi.universal.load_all_from_mysql$.main(load_all_from_mysql.scala:24)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at com.mingzhi.universal.load_all_from_mysql.main(load_all_from_mysql.scala)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at java.lang.reflect.Method.invoke(Method.java:498)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
11-08-2020 11:38:52 CST t_uac_user INFO - 	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO storage.DiskBlockManager: Shutdown hook called
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.ShutdownHookManager: Shutdown hook called
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-7596b4c8-6950-47e0-a4c3-29042fe6917d/userFiles-5e9d9769-a283-41f1-882a-e53364bea979
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-7596b4c8-6950-47e0-a4c3-29042fe6917d
11-08-2020 11:38:52 CST t_uac_user INFO - 20/08/11 11:38:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a0f37fef-d344-420e-91e2-7fa5f37490ee
11-08-2020 11:38:53 CST t_uac_user INFO - Process with id 22057 completed unsuccessfully in 4 seconds.
11-08-2020 11:38:53 CST t_uac_user ERROR - Job run failed!
java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:305)
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:813)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:602)
	at azkaban.execapp.JobRunner.run(JobRunner.java:563)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
	at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:125)
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:297)
	... 8 more
11-08-2020 11:38:53 CST t_uac_user ERROR - azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1 cause: azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 1
11-08-2020 11:38:53 CST t_uac_user INFO - Finishing job t_uac_user at 1597117133335 with status FAILED

 解决方案：

Address already in use: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consi
der explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an a
vailable port or increasing spark.port.maxRetries.

saprk-sql 启动时报错 (yarn client 模式)

描述：每一个Spark任务都会占用一个SparkUI端口，默认为4040，如果被占用则依次递增端口重试。但是有个默认重试次数，为16次。16次重试都失败后，会放弃该任务的运行。

解决

初始化SparkConf时，添加conf.set(“spark.port.maxRetries”,“100”)语句

使用spark-submit提交任务时，在命令行中添加
--conf spark.port.maxRetries=128

在spark-defaults.conf中添加spark.port.maxRetries 100
都教授2000
关注
azkaban中提醒“azkaban.jobExecutor.utils.process.ProcessFailureException”报错解答
data_curd的博客
09-09 4313
09-09-2020 15:54:24 CST test01 INFO - Starting job test01 at 1599638064795 09-09-2020 15:54:24 CST test01 INFO - job JVM args: -Dazkaban.flowid=test01 -Dazkaban.execid=22 -Dazkaban.jobid=test01 09-09-2020 15:54:24 CST test01 INFO - user.to.proxy property w
使用Azkaban过程中遇到的非常见问题以及解决办法
chengyuan54的博客
09-07 1万+
使用azkaban3.9遇到的问题 启动azkaban并且安装插件遇到的问题 启动azkaban报错1： Could not find variable substitution for variable(s) [jobtype.classpath->hadoop.classpath] 错误信息： 1) Error injecting constructor, azkaban.jobtype.JobTypeManagerException: azkaban.jobtype.JobTypeM
azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 64
qq_40348184的博客
03-20 1209
azkaban相关问题
Address already in use: Service 'SparkUI' failed after 16 retries (starting from 4040)!
onway_goahead的博客
03-25 2443
Address already in use: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consi der explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for Spark...
azkaban调度任务的时候报错：hive: command not found
jackfeng86的专栏
12-27 1853
azkaban调度任务的时候报错：hive: command not found，具体错误如下： 27-12-2021 15:06:26 CST hdfs_to_ods_log INFO - ================== 日志日期为 2021-12-25 ================== 27-12-2021 15:06:26 CST hdfs_to_ods_log INFO - /home/test/bin/hdfs_to_ods_log.sh: line 18: hive: comman
微服务之RabbitMQ消息发送失败重试（上）
Jumpingyyds的博客
03-12 6338
本博客讲述的是RabbitMQ基于AMQP协议的消息中间件，在互联网应用的开发中，RabbitMQ是确保消息发送和消费成功，设置消息有效期和延迟队列的一个重要的中间件。它的特点具有安全性、可用性、集群、多协议支持、可视化客户端等等。 关于他的详细介绍和作用就不一一赘述了，下面引用一个邮件发送的案例来加深理解。 搭建一个简单的邮件发送服务，需要用到的开发工具有IntelliJ IDEA，RabbitMQ、MySQL，Postman用来作为测试接口。 案例介绍：假如现在某公司新招聘了一批员工，需要给新员
Nginx失败重试中的HTTP协议幂等问题: non_idempotent
叉叉哥的BLOG
04-21 2392
Nginx通过反向代理做负载均衡时，如果被代理的其中一个服务发生错误或者超时的时候，通常希望Nginx自动重试其他的服务，从而实现服务的高可用性。实际上Nginx本身默认会有错误重试机制，并且可以通过proxy_next_upstream来自定义配置。 如果不了解HTTP协议以及Nginx的机制，就可能在使用过程中遇到各种各样的坑。例如服务出现了错误或超时却未重试，或者一些例如创建订单或发送短信这...
易语言判断端口开放
07-16
例如，设置适当的重试次数和间隔，避免因网络瞬时问题导致误判。此外，对于多线程环境，可能需要考虑同步问题，确保同一时间只有一个线程在执行网络通讯操作。 总之，易语言的网络通讯函数为开发者提供了便捷的端口...
java调用接口失败重调_java接口调用失败的退避策略
weixin_33121737的博客
02-13 1965
码农公社 210.net.cn210= 102410月24日一个重要的节日--码农(程序员)节退避策略简介在开发过程中我们经常会遇到调用接口失败的情况。遇到这种情况，我们有时候需要重试机制，常用的重试(退避)策略有：固定的时间间隔重试一次，最多重试N次：比如我现在一个接口调用失败了，不是立马返回失败，而是hold住线程，每隔2秒重新调下接口，最多调5次，只要其中一次成功了就直接返回。如果...
SpringCloud eureka 重试机制ribbon,熔断器Hystrix,简化远程调用Feign
花开富贵的博客
07-22 455
1.重试机制 ribbon 1.1 解释:当一次服务调用失败后，不会立即抛出异常，而是再次重试另一个服务。 1.2 实现步骤: 在服务调用端配置文件中开启重试机制 spring: cloud: loadbalancer: retry: enabled: true # 开启Spring Cloud的重试功能 service: #负载均衡规则的配置 ...
提交azkaban任务时候报错：java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureExceptio
weixin_54401020的博客
05-05 2312
1 运行azakban提交任务时候报错信息 java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException 需要把提交到网页上的所有脚本文件格式都改成unix（请注意是所有），如果你是Notepad++ 编辑的 修改方法如下： 修改重新上传即可。 ...
Windows10：spark报错。WARN Utils: Service ‘SparkUI‘ could not bind on port 4040. Attempting port 4041.
灿灿的博客
07-16 5950
最近在单机练习SPARK的过程中，遇到下面的问题： 本机：Windows10 主要问题是：我之前打开了一个shell窗口，用Scala模式运行。 然后在里面设置了spark的context，此时spark-shell里已经有一个context对象了，所以在pyspark中我新建创建的数据无法使用。 解决方法： 关闭之前打开的spark-shell. 这里我采用的是ctrl+c，退出该程序命令。 ...
【Azkaban报错解决】FAILED: SemanticException Failed to get a spark session
prague6695的博客
05-07 4642
问题描述： ods_to_dwd_log报以下错误 ods_to_dim_db报一下错误 可以看出都是相同报错，无法创建spark事务 Logging initialized using configuration in jar:file:/opt/module/hive/lib/hive-common-3.1.2.jar!/hive-log4j2.properties Async: true 05-05-2022 14:31:02 CST ods_to_dim_db INFO - Hive Sess
azkaban报错：azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 64
jackfeng86的专栏
01-05 3537
azkaban报错：azkaban.jobExecutor.utils.process.ProcessFailureException: Process exited with code 64
ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not pars
qq_53810226的博客
03-14 5896
ERROR SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Could not parse Master URL: ‘<pyspark.conf.SparkConf object at 0x7fb5da40a850>’ 报错信息 22/03/14 10:58:26 WARN NativeCodeLoader: Unable to load native-hadoop library f
单机运行Spark Shell遇到的一个低级错误
热门推荐
程裕强的专栏
04-08 2万+
bin/spark-shell下载spark-2.1.0-bin-hadoop2.7.tgz，解压缩直接进入spark根目录，然后运行bin/spark-shell即可进入。 但是今天遇到了一个低级错误： java.net.BindException: Cannot assign requested address: Service ‘sparkDriver’ failed after 16 r
spark解决报错WARN util.Utils: Service 'sparkWorker' could not bind on port 0. Attempting port 1.
一直在努力的小渣渣
05-30 7231
WARN util.Utils: Service ‘sparkWorker’ could not bind on port 0. Attempting port 1. 这个报错提示是在spark启动从节点时出现的。 解决的方法是，在spark-env.sh中加入一条 SPARK_LOCAL_IP=127.0.0.1 然后就完美解决报错了！
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值