三台节点,计划部署分配如下:
ip 主机名 用途
172.16.48.201 sg201 namenode
172.16.48.202 sg202 datanode
172.16.48.203 sg203 datanode
三台节点的etc/hosts文件设置如下:
- 172.16.48.203 sg203
- 172.16.48.202 sg202
- 172.16.48.201 sg201
关闭三台节点的防火墙 service iptables stop
一、安装基础工具1.三台节点上安装jdk并设置好环境变量:http://blog.csdn.net/chenxingzhen001/article/details/7732692
2.安装ssh服务
二、配置三台节点的ssh无密码登录
参考:http://blog.csdn.net/chenxingzhen001/article/details/7740357
三、安装并配置hadoop 1.0.3
1.在hadoop官网上下载hadoop-1.0.3.tar.gz,并解压在/opt目录下
cd /opt
tar zxf hadoop-1.0.3.tar.gz
2.修改hadoop-env.sh 配置
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_05 #设置JAVA_HOME
export HADOOP_HOME_WARN_SUPPRESS=1 #取消环境变量$HADOOP_HOME过时的警告
3.配置环境变量
vi /etc/profile
- JAVA_HOME=/usr/java/jdk1.7.0_05
- ANT_HOME=/usr/local/apache-ant-1.8.3
- HADOOP_HOME=/opt/hadoop-1.0.3
- CLASSPATH=.:$JAVA_HOME/lib/tools.jar
- PATH=$JAVA_HOME/bin:$ANT_HOME/bin:$HADOOP_HOME/bin:$PATH
- export JAVA_HOME ANT_HOME HADOOP_HOME CLASSPATH PATH
source /etc/profile
4.设置conf/masters文件内容
- vi conf/masters
- 172.16.48.201 # secondaryNameNode
- vi /conf/slaves
- 172.16.48.202 #datanode
- 172.16.48.203 #datanode
vim conf/hdfs-site.xml
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <!-- Put site-specific property overrides in this file. -->
- <configuration>
- <property>
- <name>dfs.name.dir</name>
- <value>/opt/hadoop-1.0.3/name</value>
- </property>
- <property>
- <name>dfs.data.dir</name>
- <value>/opt/hadoop-1.0.3/data</value>
- </property>
- <property>
- <name>dfs.replication</name>
- <value>2</value>
- </property>
- </configuration>
dfs.name.dir:NameNode上的本地文件路径,用于持久存储命名空间和日志信息文件等内容。该参数可以有多个值,值之间用逗号分割,表示文件映射关系信息将会被复制到每个目录中做冗余备份。
dfs.data.dir:DataNode上的本地文件路径,用于存放文件数据块。该参数可以有多个值,值之间用逗号分割,表示文件数据块将会被复制到每个目录中做冗余备份,通常配置为不同的设备
注意:这里的name、data目录不能预先创建,hadoop格式化时会自动创建,如果预先创建反而会有问题
dfs.replication:表示文件数据块要冗余备份的数量,默认值为3
7.配置文件:conf/mapred-site.xml
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
-
- <!-- Put site-specific property overrides in this file. -->
-
- <configuration>
- <property>
- <name>mapred.job.tracker</name>
- <value>172.16.48.201:9001</value>
- </property>
- </configuration>
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <!-- Put site-specific property overrides in this file. -->
- <configuration>
- <property>
- <name>mapred.job.tracker</name>
- <value>172.16.48.201:9001</value>
- </property>
- </configuration>
配置文件参数说明:
mapred.job.tracker:JobTracker(任务分配管理)的主机(或者IP)和端口
mapred.local.dir:本地文件路径,用逗号分割的路径列表,是Map/Reduce临时数据存放的地方
8.配置文件:conf/core-site.xml
- <?xml version="1.0"?>
- <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
- <!-- Put site-specific property overrides in this file. -->
- <configuration>
- <property>
- <name>fs.default.name</name>
- <value>hdfs://172.16.48.201:9000</value>
- </property>
- <property>
- <name>fs.checkpoint.period</name>
- <value>3600</value>
- </property>
- <property>
- <name>fs.checkpoint.size</name>
- <value>67108864</value>
- </property>
- <property>
- <name>hadoop.tmp.dir</name>
- <value>/opt/hadoop-1.0.3/tmp</value>
- </property>
- </configuration>
9.将布署进行分发
- [root@sg201 hadoop-1.0.3]# bin/hadoop namenode -format
- 12/07/13 11:08:58 INFO namenode.NameNode: STARTUP_MSG:
- /************************************************************
- STARTUP_MSG: Starting NameNode
- STARTUP_MSG: host = sg201/127.0.0.1
- STARTUP_MSG: args = [-format]
- STARTUP_MSG: version = 1.0.3
- STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by
- 'hortonfo' on Tue May 8 20:31:25 UTC 2012
- ************************************************************/
- 12/07/13 11:08:59 INFO util.GSet: VM type = 64-bit
- 12/07/13 11:08:59 INFO util.GSet: 2% max memory = 17.77875 MB
- 12/07/13 11:08:59 INFO util.GSet: capacity = 2^21 = 2097152 entries
- 12/07/13 11:08:59 INFO util.GSet: recommended=2097152, actual=2097152
- 12/07/13 11:08:59 INFO namenode.FSNamesystem: fsOwner=root
- 12/07/13 11:08:59 INFO namenode.FSNamesystem: supergroup=supergroup
- 12/07/13 11:08:59 INFO namenode.FSNamesystem: isPermissionEnabled=true
- 12/07/13 11:08:59 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
- 12/07/13 11:08:59 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s),
- accessTokenLifetime=0 min(s)
- 12/07/13 11:08:59 INFO namenode.NameNode: Caching file names occuring more than 10 times
- 12/07/13 11:09:00 INFO common.Storage: Image file of size 110 saved in 0 seconds.
- 12/07/13 11:09:00 INFO common.Storage: Storage directory /opt/hadoop-1.0.3/name has been successfully formatted.
- 12/07/13 11:09:00 INFO namenode.NameNode: SHUTDOWN_MSG:
- /************************************************************
- SHUTDOWN_MSG: Shutting down NameNode at sg201/127.0.0.1
- ************************************************************/
2. 启动hdfs守护进程
- [root@sg201 hadoop-1.0.3]# bin/start-dfs.sh
- starting namenode, logging to /opt/hadoop-1.0.3/libexec/../logs/hadoop-root-namenode-sg201.out
- 172.16.48.202: starting datanode, logging to /opt/hadoop-1.0.3/libexec/../logs/hadoop-root-datanode-sg202.out
- 172.16.48.203: starting datanode, logging to /opt/hadoop-1.0.3/libexec/../logs/hadoop-root-datanode-sg203.out
- root@172.16.48.201's password:
- 172.16.48.201: starting secondarynamenode, logging to /opt/hadoop-1.0.3/libexec/../logs/hadoop-root-secondarynamenode-sg201.out
3.启动mapreduce守护进程
- [root@sg201 hadoop-1.0.3]# bin/start-mapred.sh
- starting jobtracker, logging to /opt/hadoop-1.0.3/libexec/../logs/hadoop-root-jobtracker-sg201.out
- 172.16.48.203: starting tasktracker, logging to /opt/hadoop-1.0.3/libexec/../logs/hadoop-root-tasktracker-sg203.out
- 172.16.48.202: starting tasktracker, logging to /opt/hadoop-1.0.3/libexec/../logs/hadoop-root-tasktracker-sg202.out
五、验证安装是否成功
使用jps命令查看启动的守护进程有哪些:
- [root@sg201 conf]# jps
- 12560 NameNode
- 17688 Jps
- 12861 JobTracker
- 12755 SecondaryNameNode
- 5855 MyEclipse
- [root@sg203 conf]# jps
- 11732 DataNode
- 14336 Jps
- 11856 TaskTracker
另外可通过浏览器访问:
NameNode http://172.16.48.201:50070/
JobTracker http://172.16.48.201:50030/
在namenode上查看集群状态
- [root@sg201 conf]# hadoop dfsadmin -report
- Configured Capacity: 1380625408000 (1.26 TB)
- Present Capacity: 1174286331904 (1.07 TB)
- DFS Remaining: 1174286249984 (1.07 TB)
- DFS Used: 81920 (80 KB)
- DFS Used%: 0%
- Under replicated blocks: 0
- Blocks with corrupt replicas: 0
- Missing blocks: 0
- -------------------------------------------------
- Datanodes available: 2 (2 total, 0 dead)
- Name: 172.16.48.203:50010
- Decommission Status : Normal
- Configured Capacity: 545281376256 (507.83 GB)
- DFS Used: 40960 (40 KB)
- Non DFS Used: 157135388672 (146.34 GB)
- DFS Remaining: 388145946624(361.49 GB)
- DFS Used%: 0%
- DFS Remaining%: 71.18%
- Last contact: Tue Jul 17 10:25:12 CST 2012
- Name: 172.16.48.202:50010
- Decommission Status : Normal
- Configured Capacity: 835344031744 (777.97 GB)
- DFS Used: 40960 (40 KB)
- Non DFS Used: 49203687424 (45.82 GB)
- DFS Remaining: 786140303360(732.15 GB)
- DFS Used%: 0%
- DFS Remaining%: 94.11%
- Last contact: Tue Jul 17 10:25:14 CST 2012
五、关闭hadoop
关闭hdfs守护进程
- [root@sg201 hadoop-1.0.3]# bin/stop-dfs.sh
- no namenode to stop
- 172.16.48.203: no datanode to stop
- 172.16.48.202: no datanode to stop
- root@172.16.48.201's password:
- 172.16.48.201: stopping secondarynamenode
- [root@sg201 hadoop-1.0.3]# bin/stop-mapred.sh
- stopping jobtracker
- 172.16.48.203: no tasktracker to stop
- 172.16.48.202: no tasktracker to stop