1、本文目标是在linux单机环境下配置hadoop2.7.6的伪分布模式。
2、在hadoop-2.7.6/etc/hadoop目录下修改如下配置文件内容(如果没有配置文件,自己创建一个即可):
2.1、core-site.xml:<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/home/hadoop/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:8020</value>
</property>
</configuration>
2.2、hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/hadoop/hadoopinfra/hdfs/datanode</value>
</property>
</configuration>
2.3、mapred-site.xml:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
2.4、yarn-site.xml:
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
3、基于ssh实现对localhost的无密码访问:
生成无密码访问的秘钥:
ssh-keygen -t rsa -P " " -f ~/.ssh/id_rsa
保存秘钥,实现无密码访问:
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
4、启动hadoop启动进程:
1、格式化hadoop文件系统:
hadoop namenode -format
2、启动hadoop的相关进程:
cd hadoop/sbin
./start-all.sh(启动过程中遇到交互式选择,选择yes即可)
5、执行hadoop fs -copyFromLocal命令时遇到的问题:
问题1、
learn_hadoop@linux-hr0n:~> hadoop fs -copyFromLocal ./test.txt hdfs://localhost/test.txt18/05/14 14:31:36 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /test.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this operation.
......
解决办法:1、rm -rf ~/hadoopinfra/hdfs/datanode/current
2、重新格式化:hadoop namenode -format
3、使用jps查看:
9195 DataNode
20514 Jps
10261 ResourceManager
10505 NodeManager
9693 SecondaryNameNode
8848 NameNode
发现有了datanode,问题解决。
问题2、
sh hadoop-2.7.6/sbin/start-all.sh启动报错:
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
hadoop/hadoop-2.7.6/sbin/start-all.sh: 111: /home/yang/hadoop/hadoop-2.7.6/sbin/../libexec/hadoop-config.sh: Syntax error: word unexpected (expecting ")")
解决方法:
不要使用sh去执行启动脚本,进入到脚本所在的sbin目录,直接执行脚本启动:
./start-all.sh