Hadoop单机单节点集群安装配置

1. 安装jdk

sudo vim /etc/profile
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile

2. ssh无密码登录

cd ~/.ssh
ssh-keygen -t rsa
cp id_rsa.pub authorized_keys

3. 安装hadoop

使用编译好的hadoop-3.0.0-SNAPSHOT.tar.gz
cd /usr/local
sudo tar zxvf ~/hadoop-3.0.0-SNAPSHOT.tar.gz

4. 创建hadoop数据目录

sudo mkdir /var/hadoop
sudo chmod 777 /var/hadoop

5. 配置hadoop

etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
如果没配置此项，会如下错误： 
Error: JAVA_HOME is not set and could not be found.

etc/hadoop/slaves
把localhost改为ip地址

etc/hadoop/core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://ip:9100</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hadoop</value>
</property>
fs.defaultFS里要设置为本机的ip地址，不要使用localhost
不然在其他机器点击namenode web界面里的Browse the filesystem时，地址会跳到127.0.0.1

6. 使用Yarn(可选)

etc/hadoop/mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

etc/hadoop/yarn-site.xml
<property> 
<name>yarn.nodemanager.aux-services</name> 
<value>mapreduce_shuffle</value> 
</property>
如果不配置会报如下错误：
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist

7. 格式化hdfs数据目录

bin/hdfs namenode -format

8. 启动hadoop

sbin/start-dfs.sh
sbin/start-yarn.sh

9. Web管理地址

NameNode http://ip:50070
ResourceManager http://ip:8088

10. 测试

bin/hdfs dfs -mkdir /words
bin/hdfs dfs -mkdir /words/input
bin/hdfs dfs -copyFromLocal ~/words/*.txt /words/input/
bin/hadoop jar ~/hadoop-test.jar