4台机器
176.177.178,179
176是resourcemanager
177是namenode
178是datanode,nodemanager
179是datanode,nodemanager
首先为了方便,4台机器之前全部建立ssh无密码登录,详见:http://blog.csdn.net/simonchi/article/details/39054437
配好hostname到ip的映射,这里特别注意,必须得配,IP是不行的,解析不了,而且注意注释掉原来hosts文件中的其它无关内容
cat /etc/hosts
192.168.11.176 com23.authentication
192.168.11.177 com22.authentication
192.168.11.178 com21.authentication
192.168.11.179 com20.authentication
下面,我么你在176上配置所有需要配置的文件,让后将所有文件拷贝到其它机器即可。
1、JAVA_HOME内容添加
hadoop-env.sh
修改JAVA_HOME
yarn-env.sh
修改JAVA_HOME
这个就不说了
2、core-site.xml
- 属性”fs.defaultFS“表示NameNode节点地址,由”hdfs://主机名(或ip):端口号”组成。
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.11.177:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
3、hdfs-site.xml
- 属性“dfs.namenode.name.dir”表示NameNode存储命名空间和操作日志相关的元数据信息的本地文件系统目录,该项默认本地路径为”/tmp/hadoop-{username}/dfs/name”;
- 属性”dfs.datanode.data.dir“表示DataNode节点存储HDFS文件的本地文件系统目录,由”file://本地目录”组成,该项默认本地路径为”/tmp/hadoop-{username}/dfs/data”。
- 属性“dfs.namenode.secondary.http-address”表示SecondNameNode主机及端口号(如果无需额外指定SecondNameNode角色,可以不进行此项配置);
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/namenodedir,/home/hadoop/namenodedirbak</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/datanodedir,/home/hadoop/datanodedirbak</value>
</property>
4、yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>192.168.11.176:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>192.168.11.176:18030</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>192.168.11.176:18088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>192.168.11.176:8025</value>
</property>
5、mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1536</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>192.168.11.176:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>192.168.11.176:19888</value>
</property>
6、slaves
com21.authentication
com20.authentication
安装完毕后,需登陆NameNode节点或任一DataNode节点执行hdfs namenode -format格式化集群HDFS文件系统;
注:如果非第一次格式化HDFS文件系统,则需要在进行格式化操作前分别将NameNode和各个DataNode节点的dfs.namenode.name.dir目录下的所有内容清空。
好了,拷贝到其它机器。
7、启动hadoop集群
1、格式化分布式文件系统
sh bin/hdfs namenode -format
|
2、在177上启动namenode节点
sh sbin/hadoop-daemon.sh --config etc/hadoop --script hdfs start namenode
|
3、启动slaves中的datanode节点
sh sbin/hadoop-daemon.sh --config etc/hadoop/ --script hdfs start datanode
|
4、在176上启动resourcemanager
sh sbin/yarn-daemon.sh --config etc/hadoop/ start resourcemanager
|
5、启动slaves中的nodemanager节点
sh sbin/yarn-daemon.sh --config etc/hadoop/ start nodemanager
|
http://192.168.11.177:50070 观察集群情况
176
177
178
179
|
8、关闭hadoop集群
1、停止namenode
sh sbin/hadoop-daemon.sh --config etc/hadoop/ --script hdfs stop namenode
|
2、停止datanode
sh sbin/hadoop-daemon.sh --config etc/hadoop/ --script hdfs stop datanode
|
3、停止resourcemanager
sh sbin/yarn-daemon.sh --config etc/hadoop/ stop resourcemanager
|
4、停止nodemanager
sh sbin/yarn-daemon.sh --config etc/hadoop/ stop nodemanager
|
9、文件系统命令的使用
详见: http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/FileSystemShell.html#mkdir
1>创建文件夹
./hdfs dfs -mkdir -p /user/hadoop/dir1 /user/hadoop/dir2
[root@com22 bin]# ./hdfs dfs -ls -R /
15/01/26 16:47:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
drwxr-xr-x - root supergroup 0 2015-01-26 16:43 /user
drwxr-xr-x - root supergroup 0 2015-01-26 16:43 /user/hadoop
drwxr-xr-x - root supergroup 0 2015-01-26 16:43 /user/hadoop/dir1
drwxr-xr-x - root supergroup 0 2015-01-26 16:43 /user/hadoop/dir2
警告哪一行的问题,必须自己用下载hadoop源码,编译成64位的来解决。
2>传文件到hdfs
./hdfs dfs -put /home/hadoop/hadoop-2.6.0/etc/hadoop/ /user/hadoop/dir1
[root@com22 bin]# ./hdfs dfs -ls -R /user/hadoop/dir1/hadoop
15/01/26 17:08:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-rw-r--r-- 3 root supergroup 4436 2015-01-26 17:08 /user/hadoop/dir1/hadoop/capacity-scheduler.xml
-rw-r--r-- 3 root supergroup 1335 2015-01-26 17:08 /user/hadoop/dir1/hadoop/configuration.xsl
-rw-r--r-- 3 root supergroup 318 2015-01-26 17:08 /user/hadoop/dir1/hadoop/container-executor.cfg
-rw-r--r-- 3 root supergroup 994 2015-01-26 17:08 /user/hadoop/dir1/hadoop/core-site.xml
-rw-r--r-- 3 root supergroup 3670 2015-01-26 17:08 /user/hadoop/dir1/hadoop/hadoop-env.cmd
-rw-r--r-- 3 root supergroup 4256 2015-01-26 17:08 /user/hadoop/dir1/hadoop/hadoop-env.sh
-rw-r--r-- 3 root supergroup 2490 2015-01-26 17:08 /user/hadoop/dir1/hadoop/hadoop-metrics.properties
-rw-r--r-- 3 root supergroup 2598 2015-01-26 17:08 /user/hadoop/dir1/hadoop/hadoop-metrics2.properties
-rw-r--r-- 3 root supergroup 9683 2015-01-26 17:08 /user/hadoop/dir1/hadoop/hadoop-policy.xml
-rw-r--r-- 3 root supergroup 1072 2015-01-26 17:08 /user/hadoop/dir1/hadoop/hdfs-site.xml
-rw-r--r-- 3 root supergroup 1449 2015-01-26 17:08 /user/hadoop/dir1/hadoop/httpfs-env.sh
-rw-r--r-- 3 root supergroup 1657 2015-01-26 17:08 /user/hadoop/dir1/hadoop/httpfs-log4j.properties
-rw-r--r-- 3 root supergroup 21 2015-01-26 17:08 /user/hadoop/dir1/hadoop/httpfs-signature.secret
-rw-r--r-- 3 root supergroup 620 2015-01-26 17:08 /user/hadoop/dir1/hadoop/httpfs-site.xml
-rw-r--r-- 3 root supergroup 3523 2015-01-26 17:08 /user/hadoop/dir1/hadoop/kms-acls.xml
-rw-r--r-- 3 root supergroup 1325 2015-01-26 17:08 /user/hadoop/dir1/hadoop/kms-env.sh
-rw-r--r-- 3 root supergroup 1631 2015-01-26 17:08 /user/hadoop/dir1/hadoop/kms-log4j.properties
-rw-r--r-- 3 root supergroup 5511 2015-01-26 17:08 /user/hadoop/dir1/hadoop/kms-site.xml
-rw-r--r-- 3 root supergroup 11291 2015-01-26 17:08 /user/hadoop/dir1/hadoop/log4j.properties
-rw-r--r-- 3 root supergroup 938 2015-01-26 17:08 /user/hadoop/dir1/hadoop/mapred-env.cmd
-rw-r--r-- 3 root supergroup 1383 2015-01-26 17:08 /user/hadoop/dir1/hadoop/mapred-env.sh
-rw-r--r-- 3 root supergroup 4113 2015-01-26 17:08 /user/hadoop/dir1/hadoop/mapred-queues.xml.template
-rw-r--r-- 3 root supergroup 1223 2015-01-26 17:08 /user/hadoop/dir1/hadoop/mapred-site.xml
-rw-r--r-- 3 root supergroup 758 2015-01-26 17:08 /user/hadoop/dir1/hadoop/mapred-site.xml.template
-rw-r--r-- 3 root supergroup 42 2015-01-26 17:08 /user/hadoop/dir1/hadoop/slaves
-rw-r--r-- 3 root supergroup 2316 2015-01-26 17:08 /user/hadoop/dir1/hadoop/ssl-client.xml.example
-rw-r--r-- 3 root supergroup 2268 2015-01-26 17:08 /user/hadoop/dir1/hadoop/ssl-server.xml.example
-rw-r--r-- 3 root supergroup 2237 2015-01-26 17:08 /user/hadoop/dir1/hadoop/yarn-env.cmd
-rw-r--r-- 3 root supergroup 4606 2015-01-26 17:08 /user/hadoop/dir1/hadoop/yarn-env.sh
-rw-r--r-- 3 root supergroup 1351 2015-01-26 17:08 /user/hadoop/dir1/hadoop/yarn-site.xml
后续来说说如何开发mapreduce了。。。