多个shark进程访问同一路径会出现死锁进程吊死,在spark的DAGScheduler.runJob中,submitJob后的等待。
问题解决前,使用需避免冲突,做进程监控。
jstack打印的堆栈(spark0.9.1):
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c0586048> (a org.apache.spark.scheduler.JobWaiter)
at java.lang.Object.wait(Object.java:485)
at org.apache.spark.scheduler.JobWaiter.awaitResult(JobWaiter.scala:73)
- locked <0x00000000c0586048> (a org.apache.spark.scheduler.JobWaiter)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:472)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:852)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:870)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:891)
at shark.execution.FileSinkOperator.execute(FileSinkOperator.scala:166)
at shark.execution.SparkTask.execute(SparkTask.scala:99)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1362)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1146)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:952)
at shark.SharkCliDriver.processCmd(SharkCliDriver.scala:338)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
at shark.SharkCliDriver$.main(SharkCliDriver.scala:171)
at shark.SharkCliDriver.main(SharkCliDriver.scala)