部署hadoop2.6.0

1. 下载 hadoop2.6.0
2. 解压tar -xvf hadoop-2.6.0.tar.gz
3. 复制到/usr/local sudo cp -r hadoop-2.6.0 /usr/local/hadoop
4. 修改目录归属为hadoop sudo chown -R hadoop:users /usr/local/hadoop/
5. 编辑hadoop配置文件

core-site.xml
hdfs-site.xml
mapred-site.xml
yarn-site.xml
slaves

6. 配置core-site.xml

<configuration>
	<property>
		<name>fs.default.name</name>
		<value>hdfs://master:9000</value>
	</property>
	<property>
		<name>hadoop.tmp.dir</name>
		<value>file:/home/hadoop/tmp</value>
	</property>
	<property>
		<name>fs.trash.interval</name>
		<value>1440</value>
	</property>
</configuration>

7. 配置hdfs-site.xml

<configuration>
　　<property>
　　　　<name>dfs.replication</name>
　　　　<value>2</value>
　　</property>
　　<property>
　　　　<name>dfs.namenode.name.dir</name>
　　　　<value>file:/home/hadoop/tmp/dfs/name</value>
　　</property>
　　<property>
　　　　<name>dfs.datanode.data.dir</name>
　　　　<value>file:/home/hadoop/tmp/dfs/data</value>
　　</property>
　　<property>
　　　　<name>dfs.namenode.http-address</name>
　　　　<value>master:50070</value>
　　</property>
　　<property>
　　　　<name>dfs.namenode.secondary.http-address</name>
　　　　<value>master:50090</value>
　　</property>
</configuration>

8. 配置mapred-site.xml

<configuration>
	<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.address</name>
		<value>master:10020</value>
		<description>MapReduce JobHistory Server IPC host:port</description>
	</property>
	<property>
		<name>mapreduce.jobhistory.webapp.address</name>
		<value>master:19888</value>
		<description>MapReduce JobHistory Server Web UI host:port</description>
	</property>
	<property>
		<name>mapreduce.jobhistory.done-dir</name>
		<value>/history/done</value>
	</property>
	<property>
		<name>mapreduce.jobhistory.intermediate-done-dir</name>
		<value>/history/done_intermediate</value>
	</property>
	<property>
		<name>mapreduce.map.memory.mb</name>
		<value>2048</value>
	</property>
	<property>
		<name>mapreduce.reduce.memory.mb</name>
		<value>2048</value>
	</property>
	<property>
		<name>mapreduce.map.java.opts</name>
		<value>-Xmx2304m</value>
	</property>
	<property>
		<name>mapreduce.reduce.java.opts</name>
		<value>-Xmx2304m</value>
	</property>
	<property>
		<name>yarn.scheduler.minimum-allocation-mb</name>
		<value>1024</value>
	</property>
	<property>
		<name>yarn.scheduler.maximum-allocation-mb</name>
		<value>3072</value>
	</property>
</configuration>

9. 配置yarn-site.xml

<configuration>
	<property>
		<name>yarn.resourcemanager.hostname</name>
		<value>master</value>
	</property>
	<property>
		<name>yarn.nodemanager.aux-services</name>
		<value>mapreduce_shuffle</value>
	</property>
	<property>
		<name>yarn.log-aggregation-enable</name>
		<value>true</value>
	</property>
	<property>
		<name>yarn.log-aggregation.retain-seconds</name>
		<value>604800</value>
	</property>
</configuration>

10. 配置slaves

slave1
slave2

11. 配置~/.bashrc中环境变量

HADOOP_HOME=/usr/local/hadoop
PATH=$JAVA_HOME/bin:$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

12. 将hadoop目录以及.bashrc同步到从节点

hadoop@master:~> scp -r /usr/local/hadoop/ root@slave1:/usr/local
hadoop@master:~> scp -r /usr/local/hadoop/ root@slave2:/usr/local
hadoop@master:~> scp ~/.bashrc hadoop@slave1:/home/hadoop
hadoop@master:~> scp ~/.bashrc hadoop@slave2:/home/hadoop

13. 修改从节点hadoop目录权限

hadoop@slave1:~> sudo chown -R hadoop:users /usr/local/hadoop/
hadoop@slave2:~> sudo chown -R hadoop:users /usr/local/hadoop/

14. 格式化HDFS

hadoop@master:~> hadoop namenode -format
0/07/11 10:16:33 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master/192.168.88.100
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.0
...
20/07/11 10:16:34 INFO namenode.NNConf: ACLs enabled? false
20/07/11 10:16:34 INFO namenode.NNConf: XAttrs enabled? true
20/07/11 10:16:34 INFO namenode.NNConf: Maximum size of an xattr: 16384
20/07/11 10:16:34 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1430712971-192.168.88.100-1594433794505
20/07/11 10:16:34 INFO common.Storage: Storage directory /home/hadoop/tmp/dfs/name has been `successfully formatted`.
20/07/11 10:16:34 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
20/07/11 10:16:34 INFO util.ExitUtil: Exiting with status 0
20/07/11 10:16:34 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.88.100
************************************************************/

15. 启动HDFS

hadoop@master:/usr/local/hadoop/sbin> ./start-dfs.sh
20/07/11 10:23:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [master]
master: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-master.out
slave1: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave1.out
slave2: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hadoop-datanode-slave2.out
Starting secondary namenodes [master]
master: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-master.out
20/07/11 10:23:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16. 启动YARN

hadoop@master:/usr/local/hadoop/sbin> ./start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-resourcemanager-master.out
slave2: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave2.out
slave1: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hadoop-nodemanager-slave1.out

17. 停止Hadoop命令

/usr/local/hadoop/sbin/stop-yarn.sh
/usr/local/hadoop/sbin/stop-dfs.sh