关闭

pig并发执行的错误

标签: pighadoophdfs
564人阅读 评论(0) 收藏 举报
分类:

错误信息为:

ERROR 2017: Internal error creating job configuration.

org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:859)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:296)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:190)
        at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
        at org.apache.pig.PigServer.execute(PigServer.java:1297)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
        at org.apache.pig.PigServer.executeBatch(PigServer.java:353)
        at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
        at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
        at com.bonc.dataplatform.bdi.dataflow.monitor.Mains.run(Mains.java:573)
        at com.bonc.dataplatform.bdi.dataflow.monitor.Mains.run(Mains.java:123)
        at com.bonc.dataplatform.bdi.dataflow.monitor.Execute.execute(Execute.java:79)
        at com.bonc.dataplatform.bdi.dataflow.DataflowTask.runScript(DataflowTask.java:233)
        at com.bonc.dataplatform.bdi.dataflow.DataflowTask.pigExecute(DataflowTask.java:203)
        at com.bonc.dataplatform.bdi.dataflow.DataflowTask.run(DataflowTask.java:127)
        at com.bonc.dataplatform.bdi.taskmanager.task.manager.TaskExecuteCommand.run(TaskExecuteCommand.java:87)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Filesystem closed
        at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:703)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1755)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1124)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120) 
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1398)
        at org.apache.pig.backend.hadoop.datastorage.HPath.exists(HPath.java:106)
        at org.apache.pig.impl.io.FileLocalizer.getTemporaryPath(FileLocalizer.java:480)
        at org.apache.pig.impl.io.FileLocalizer.getTemporaryPath(FileLocalizer.java:474)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.shipToHDFS(JobControlCompiler.java:1521)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.putJarOnClassPathThroughDistributedCache(JobControlCompiler.java:1498)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:513)
        ... 21 more

解决办法

修改配置文件中的 hdfs-site.xml的以下属性 (没有就加上)

<property>
 <name>fs.hdfs.impl.disable.cache</name>
 <value>true</value>
 </property>

0
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:8577次
    • 积分:141
    • 等级:
    • 排名:千里之外
    • 原创:6篇
    • 转载:0篇
    • 译文:0篇
    • 评论:0条
    文章分类
    文章存档