hadoop非HA安装，解决defalutFS（eg：beh）问题

报错如下：

（1）修改core-site.xml :

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://beh/</value>
  </property>

为如下：（因为不修改，hdfs dfs -ls / 会无法查看，报错UnknownHostException: beh）

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://****:9000/</value>
  </property>

修改为如上之后（hdfs dfs -ls / 可以查看，但是运行yarn 测试样例 wordcount无法得到yarn资源，且任务所在节点的nodemanager直接挂掉，查看nodemanager日志，发现如下报错）

注：wordcount测试样例：hadoop jar /opt/beh/core/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.13.1.jar wordcount /wkz/in/hadooptest /wkz/out

FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.IllegalArgumentException: java.net.UnknownHostException: beh
        at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406)
        at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)
        at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:735)
        at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:678)
        at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:158)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2816)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.g
etFileSystem(LogAggregationService.java:181)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.v
erifyAndCreateRemoteLogDir(LogAggregationService.java:188)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.i
nitApp(LogAggregationService.java:336)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.h
andle(LogAggregationService.java:463)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.h
andle(LogAggregationService.java:68)
        at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:182)
        at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:109)
        at java.lang.Thread.run(Thread.java:748)                                          
Caused by: java.net.UnknownHostException: beh
        ... 20 more        

（2）出现如上错误后，还原core-site.xml 为：

<property>
        <name>fs.defaultFS</name>
        <value>hdfs://beh/</value>
  </property>

运行wordcount测试样例：

hadoop jar /opt/beh/core/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.13.1.jar wordcount hdfs://****:9000/wkz/in/hadooptest     hdfs://****:9000/wkz/out

直接报错，报错找不到截图了，只能大概描述下：

第一：还是有java.net.UnknownHostException: beh

第二：mapreduce.framework.name错误

后来做了如下修改：（去掉beh有关的所有，东西，eg：dfs.namenode.rpc-address.beh变为dfs.namenode.rpc-address）

hdfs-site.xml

<configuration>
<!--
<property>
        <name>dfs.nameservices</name>
        <value>beh</value>
    </property>
-->  
<property>
        <name>dfs.namenode.rpc-address</name>
        <value>*****9000</value>
    </property>

    <property>
        <name>dfs.namenode.http-address</name>
        <value>******:50070</value>
    </property>

 <property>
        <name>dfs.namenode.name.dir</name>
        <value>/data01/dsjdev/testname</value>
</property>
 <property>
        <name>dfs.datanode.data.dir</name>
        <value>/data01/dsjdev/testdn</value>
    </property>
 <property>
        <name>dfs.block.size</name>
        <value>134217728</value>
    </property>

    <property>
        <name>dfs.datanode.handler.count</name>
        <value>20</value>
    </property>

    <property>
        <name>dfs.namenode.handler.count</name>
        <value>20</value>
    </property>

    <property>
        <name>dfs.datanode.max.xcievers</name>
        <value>131072</value>
    </property>
<property>
        <name>dfs.datanode.socket.write.timeout</name>
        <value>0</value>
    </property>

    <property>
        <name>dfs.socket.timeout</name>
        <value>180000</value>
    </property>
<property>
        <name>dfs.replication</name>
        <value>2</value>
</property>

<property>  
   <name>mapred.child.env</name>  
      <value>JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native</value>  
</property> 
</configuration>

core-site.xml

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://******:9000/</value>
  </property>
 <property>
        <name>hadoop.tmp.dir</name>
        <value>/data01/dsjdev/testnn</value>
  </property>
  <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
  </property>
  <property>
       <name>io.compression.codecs</name>
 <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value>
  </property>
 <property> 
    <name>fs.trash.interval</name> 
    <value>10080</value> 
    <description>Number of minutes between trash checkpoints. 
    If zero, the trash feature is disabled. 
    </description> 
  </property>
</configuration>

yarn-site.xml

<configuration>
 <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/data01/dsjdev/testnn_local_dir</value>
  </property>
<property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/data01/dsjdev/testnn_log_dir</value>
  </property>
 <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property> 
 <property>
    <description>Where to aggregate logs</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>hdfs://******:9000/var/log/hadoop-yarn/apps</value>
  </property>
<property>
    <name>yarn.resourcemanager.connect.retry-interval.ms</name>
    <value>2000</value>
  </property>
 <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>beh</value>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
  </property>

<property>
    <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
    <value>5000</value>
  </property>
<property>
    <name>yarn.resourcemanager.address</name>
    <value>******:23140</value>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>******:23130</value>
  </property>

  <property>
    <name>yarn.resourcemanager.webapp.https.address</name>
    <value>******:23189</value>
  </property>

  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>******:23188</value>
  </property>

  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>******:23125</value>
  </property>

  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>******:23141</value>
  </property>

<property>
    <description>Address where the localizer IPC is.</description>
    <name>yarn.nodemanager.localizer.address</name>
    <value>0.0.0.0:23344</value>
  </property>

  <property>
    <description>NM Webapp address.</description>
    <name>yarn.nodemanager.webapp.address</name>
    <value>0.0.0.0:23999</value>
  </property>

  <property>
    <name>mapreduce.shuffle.port</name>
    <value>23080</value>
  </property>

 <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
        <value>10240</value>
   </property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
    <value>6144</value>
  </property>
  <property>
      <name>yarn.nodemanager.resource.cpu-vcores</name>
          <value>2</value>
  <description>number of all the cpu cores</description>
  </property>
</configuration>

mapred-site.xml

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
<property>
    <name>mapreduce.jobhistory.address</name>
    <value>*****:10020</value>
  </property>
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>*****:19888</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.staging-dir</name>
    <value>/data01/dsjdev/hadoop-yarn/staging</value>
  </property>
<property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx2g</value>
  </property>

<property>
    <name>io.sort.mb</name>
    <value>512</value>
  </property>

<property>
    <name>io.sort.factor</name>
    <value>20</value>
  </property>
<property>
    <name>mapred.job.reuse.jvm.num.tasks</name>
    <value>-1</value>
  </property>
<property>
    <name>mapreduce.reduce.shuffle.parallelcopies</name>
    <value>20</value>
  </property>
<property>
  <name>mapred.output.compress</name>
  <value>true</value>
</property>
<property>
  <name>mapred.output.compression.codec</name>
  <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
  <name>mapreduce.map.output.compress</name>
  <value>true</value>
</property>
<property>
  <name>mapreduce.map.output.compress.codec</name>
  <value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
  <name>mapred.output.compression.type</name>
  <value>BLOCK</value>
</property>
<property>
        <name>mapreduce.cluster.local.dir</name>
        <value>/data01/dsjdev/testmapred/local</value>  
  </property>
</configuration>

修改后重启hadoop，一切正常