一 、 服务器准备阶段
由于我是用两个账号买的腾讯云服务器，并且其中两台服务器位置在上海，一台在北京。因此需要了解腾讯云私有网络文档，详情可参考腾讯云vpc文档。
如使用虚拟机搭建，直接将hosts文件配置为内网IP即可。
服务器配置，1C 2G
二、 3台节点，修改服务器/etc/hosts文件（核心操作）
[root@h201 ~]vi /etc/hosts
本机使用内网IP h201
其他主机使用外网IP h202
其他主机使用外网IP h203
[root@h202 ~]vi /etc/hosts
其他主机使用外网IP h201
本机使用内网IP h202
其他主机使用外网IP h203
[root@h203 ~]vi /etc/hosts
其他主机使用外网IP h201
其他主机使用外网IP h202
本机使用内网IP h203三、3台节点分别创建hadoop用户和组
[root@h201 ~]$ groupadd hadoop
[root@h201 ~]$ useradd hadoop -g hadoop
[root@h201 ~]$ passwd hadoop # 设置为123456四、3台节点分别安装JDK
# 使用的java版本为1.7
[root@h201 ~]$ tar -zxvf jdk-7u25-linux-i586.tar.gz -C /usr/
[root@h201 ~]$ yum install glibc.i686 # 更新glibc五、配置环境变量并重启服务器
[root@h201 ~]$ vi /etc/profile
export JAVA_HOME=/usr/jdk1.7.0_25
export JAVA_BIN=/usr/jdk1.7.0_25/bin
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
[root@h201 ~]$ reboot # source /etc/profile六、安装ssh证书，hadoop用户免密ssh
[root@h201 ~]$ su hadoop #切换至hadoop用户
[hadoop@h20x ~]$ ssh-keygen -t rsa #三台节点都输入
[hadoop@h20x ~]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub h201 #三台节点都输入
[hadoop@h20x ~]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub h202 #三台节点都输入
[hadoop@h20x ~]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub h203 #三台节点都输入七、主节点（h201）安装hadoop-2.6.0-cdh5.5.2
[root@h201 ~]$ tar -zxvf hadoop-2.6.0-cdh5.5.2.tar.gz -C /usr/local/八、修改hadoop配置文件
8.1 编辑core-site.xml
[root@h201 ~]$ vi core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://h201:9000</value>
<description>NameNode URI.</description>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
<description>Size of read/write buffer used inSequenceFiles.</description>
</property>
</configuration>8.2 编辑hdfs-site.xml
[root@h201 ~]$ cd /usr/local/hadoop-2.6.0-cdh5.5.2
[root@h201 hadoop-2.6.0-cdh5.5.2]$ mkdir -p dfs/name
[root@h201 hadoop-2.6.0-cdh5.5.2]$ mkdir -p dfs/data
[root@h201 hadoop-2.6.0-cdh5.5.2]$ mkdir -p dfs/namesecondary
[root@h201 hadoop-2.6.0-cdh5.5.2]$ cd etc/hadoop
[root@h201 hadoop]$ vi hdfs-site.xml
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>h201:50090</value>
<description>The secondary namenode http server address andport.</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///usr/local/hadoop-2.6.0-cdh5.5.2/dfs/name</value>
<description>Path on the local filesystem where the NameNodestores the namespace and transactions logs persistently.</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///usr/local/hadoop-2.6.0-cdh5.5.2/dfs/data</value>
<description>Comma separated list of paths on the local filesystemof a DataNode where it should store its blocks.</description>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///usr/local/hadoop-2.6.0-cdh5.5.2/dfs/namesecondary</value>
<description>Determines where on the local filesystem the DFSsecondary name node should store the temporary images to merge. If this is acomma-delimited list of directories then the image is replicated in all of thedirectories for redundancy.</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
8.3 编辑mapred-site.xml
[root@h201 hadoop]$ cp mapred-site.xml.template mapred-site.xml
[root@h201 hadoop]$ vi mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Theruntime framework for executing MapReduce jobs. Can be one of local, classic or yarn.</description>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>h201:10020</value>
<description>MapReduce JobHistoryServer IPC host:port</description>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>h201:19888</value>
<description>MapReduce JobHistoryServer Web UI host:port</description>
</property>
--------------------------------------------注---------------------------------------------
*****
属性”mapreduce.framework.name“表示执行mapreduce任务所使用的运行框架，默认为local，需要将其改为”yarn”
*****8.4 编辑yarn-site.xml
[root@h201 hadoop]$ vi yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>h201</value>
<description>The hostname of theRM.</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>Shuffle service that needs to be set for Map Reduceapplications.</description>
</property>8.5 编辑hadoop-env.sh
[root@h201 hadoop]$ vi hadoop-env.sh
添加 ：
export JAVA_HOME=/usr/jdk1.7.0_258.6 编辑slaves
[root@h201 hadoop]$ vi slaves
h202
h203九、将主节点的hadoop传送给另外两台服务器，并授权
# 传送
[root@h201 ~]$ scp -r /usr/local/hadoop-2.6.0-cdh5.5.2/ root@h202:/usr/local/
[root@h201 ~]$ scp -r /usr/local/hadoop-2.6.0-cdh5.5.2/ root@h203:/usr/local/
# 授权
[root@h201 ~]$chown hadoop.hadoop /usr/local/hadoop-2.6.0-cdh5.5.2/ -R
[root@h202 ~]$chown hadoop.hadoop /usr/local/hadoop-2.6.0-cdh5.5.2/ -R
[root@h203 ~]$chown hadoop.hadoop /usr/local/hadoop-2.6.0-cdh5.5.2/ -R十、主节点切换hadoop用户，声明hadoop所需环境变量
[root@h201 ~]$ su - hadoop
[hadoop@h201 ~]$ vi .bash_profile
export JAVA_HOME=/usr/jdk1.7.0_25
export JAVA_BIN=/usr/jdk1.7.0_25/bin
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export JAVA_HOME JAVA_BIN PATH CLASSPATH
HADOOP_HOME=/usr/local/hadoop-2.6.0-cdh5.5.2
HADOOP_BIN=/usr/local/hadoop-2.6.0-cdh5.5.2/bin
HADOOP_SBIN=/usr/local/hadoop-2.6.0-cdh5.5.2/sbin
HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
PATH=$HADOOP_HOME/bin:$PATH
export HADOOP_HOME HADOOP_CONF_DIR PATH
[hadoop@h201 ~]$ source .bash_profile十一、格式化namenode
11.1 在log4j日志中去除告警信息(最好三台都修改)
vim /usr/local/hadoop-2.6.0-cdh5.5.2/etc/hadoop/log4j.properties
- 添加：
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR11.2 格式化
[hadoop@h201 hadoop-2.6.0]$ bin/hadoop namenode -format
17/05/09 06:16:02 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = h101/192.168.16.101
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2-cdh3u5
STARTUP_MSG: build = git://ubuntu-slave02/var/lib/jenkins/workspace/CDH3u5-Full-RC/build/cdh3/hadoop20/0.20.2-cdh3u5/source -r 30233064aaf5f2492bc687d61d72956876102109; compiled by 'jenkins' on Fri Oct 5 17:21:34 PDT 2012
************************************************************/
Re-format filesystem in /tmp/hadoop-hadoop/dfs/name ? (Y or N) Y
17/05/09 06:16:04 INFO util.GSet: VM type = 32-bit
17/05/09 06:16:04 INFO util.GSet: 2% max memory = 19.33375 MB
17/05/09 06:16:04 INFO util.GSet: capacity = 2^22 = 4194304 entries
17/05/09 06:16:04 INFO util.GSet: recommended=4194304, actual=4194304
17/05/09 06:16:05 INFO namenode.FSNamesystem: fsOwner=hadoop (auth:SIMPLE)
17/05/09 06:16:05 INFO namenode.FSNamesystem: supergroup=supergroup
17/05/09 06:16:05 INFO namenode.FSNamesystem: isPermissionEnabled=true
17/05/09 06:16:05 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=1000
17/05/09 06:16:05 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
17/05/09 06:16:05 INFO common.Storage: Image file of size 112 saved in 0 seconds.
17/05/09 06:16:06 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
17/05/09 06:16:06 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at h101/192.168.16.101
************************************************************/十二、启动hadoop，并校验节点进程
12.1 启动
[hadoop@h201 hadoop-2.6.0]$ sbin/start-all.sh 12.2 校验
[hadoop@h201 hadoop-2.6.0]$ jps
7054 SecondaryNameNode
7844 Jps
7318 NameNode
7598 ResourceManager
[hadoop@h202 hadoop-2.6.0]$ jps
3690 DataNode
5014 Jps
4987 NodeManager
搭建结束 ~
1994
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
