Error launching job , bad input path

python /home/hduser/Workspace/Ch5_Py_mrjob_PegasosSVM/mrSVM.py -r hadoop < Workspace/Ch5_Py_mrjob_PegasosSVM/kickStart.txt

之前顺利执行过一次,也尝试过格式话namenode;重新删除建立data文件夹,问题仍然存在:
报错:

No handlers could be found for logger "mrjob.launch"
No configs found; falling back on auto-configuration
No configs specified for hadoop runner
Looking for hadoop binary in /usr/local/hadoop/bin...
Found hadoop binary: /usr/local/hadoop/bin/hadoop
Using Hadoop version 2.7.5
Looking for Hadoop streaming jar in /usr/local/hadoop...
Found Hadoop streaming jar: /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.5.jar
Creating temp directory /tmp/mrSVM.hduser.20180604.015226.736070
reading from STDIN
Copying local files to hdfs:///user/hduser/tmp/mrjob/mrSVM.hduser.20180604.015226.736070/files/...
Running step 1 of 2...
  session.id is deprecated. Instead, use dfs.metrics.session-id
  Initializing JVM Metrics with processName=JobTracker, sessionId=
  Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
  Cleaning up the staging area file:/usr/local/hadoop/hadoop_data/mapred/staging/hduser854512016/.staging/job_local854512016_0001
  Error launching job , bad input path : File does not exist: /usr/local/hadoop/hadoop_data/mapred/staging/hduser854512016/.staging/job_local854512016_0001/files/mrSVM.py#mrSVM.py
  Streaming Command Failed!
Attempting to fetch counters from logs...
Can't fetch history log; missing job ID
No counters found
Scanning logs for probable cause of failure...
Can't fetch history log; missing job ID
Can't fetch task logs; missing application ID
Step 1 of 2 failed: Command '['/usr/local/hadoop/bin/hadoop', 'jar', '/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.7.5.jar', '-files', 'hdfs:///user/hduser/tmp/mrjob/mrSVM.hduser.20180604.015226.736070/files/mrSVM.py#mrSVM.py,hdfs:///user/hduser/tmp/mrjob/mrSVM.hduser.20180604.015226.736070/files/mrjob.zip#mrjob.zip,hdfs:///user/hduser/tmp/mrjob/mrSVM.hduser.20180604.015226.736070/files/setup-wrapper.sh#setup-wrapper.sh', '-input', 'hdfs:///user/hduser/tmp/mrjob/mrSVM.hduser.20180604.015226.736070/files/STDIN', '-output', 'hdfs:///user/hduser/tmp/mrjob/mrSVM.hduser.20180604.015226.736070/step-output/0000', '-mapper', 'sh -ex setup-wrapper.sh python mrSVM.py --step-num=0 --mapper', '-reducer', 'sh -ex setup-wrapper.sh python mrSVM.py --step-num=0 --reducer']' returned non-zero exit status 512

解决办法:
在网上查询得到:在master和各个节点执行以下操作:
core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
 <property>
      <name>hadoop.tmp.dir</name>
      <value>file:///usr/local/hadoop/hadoop_data</value>
      <description>A base for other temporary directories.</description>
    </property>
</configuration>

hdfs-site.xml


<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
 <property>
      <name>hadoop.tmp.dir</name>
    <value>file:///usr/local/hadoop/hadoop_data</value>
      <description>A base for other temporary directories.</description>
    </property>
</configuration>

mapred-site.xml


<configuration>
  <property> 
    <name>mapreduce.framework.name</name> 
    <value>yarn</value> 
  </property>
</configuration>

yarn-site.xml


<configuration>

    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>128</value>
        <description>Minimum limit of memory to allocate to each container request at the Resource Manager.</description>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>2048</value>
        <description>Maximum limit of memory to allocate to each container request at the Resource Manager.</description>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
        <description>The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum.</description>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>2</value>
        <description>The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value.</description>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>4096</value>
        <description>Physical memory, in MB, to be made available to running containers</description>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>4</value>
        <description>Number of CPU cores that can be allocated for containers.</description>
    </property>

    <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value>
      <description>shuffle service that needs to be set for Map Reduce to run </description>
    </property>
</configuration>

然后删除原有的hdfs目录下的节点文件夹,新建,授权,执行
hadoop namenode -format 格式化namenode HDFS

执行start-all.sh
执行操作语句

  • 1
    点赞
  • 1
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值