Hadoop集群搭建
Hadoop版本—2.6.0稳定版
所需软件
Linux和Windows所需软件包括:
1. JavaTM1.5.x,必须安装,建议选择Sun公司发行的Java版本。
2. ssh 必须安装并且保证 sshd一直运行,以便用Hadoop 脚本管理远端Hadoop守护进程。
Windows下的附加软件需求
1. Cygwin - 提供上述软件之外的shell支持。
安装软件
安装ssh,rsync及其他依赖
yum -yinstall openssh-clients gcc-c++ vim make gcc unzip wget subversion openssh zlib zlib-devel rsync
rsync –version
解压
tar -zxvf hadoop-2.6.0.tar.gz
单机版本安装测试
编辑etc/hadoop/hadoop-env.sh
#set to the root of your Java installation
export JAVA_HOME=/usr/java/latest
# Assuming your installation directory is/usr/local/hadoop
export HADOOP_PREFIX=/usr/local/hadoop
执行bin/hadoop
显示帮助文档
单机操作
$ mkdir input
$ cp etc/hadoop/*.xml input
$ bin/hadoop jarshare/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output'dfs[a-z.]+'
$ cat output/*
$ bin/hadoop jarshare/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep input output1'con[a-z.]+'
上面的操作是打包conf里的配置作为input然后根据正则表达式去匹配,将结果集写入output.
单机伪分布式模式测试
配置
etc/hadoop/core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
etc/hadoop/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
设置ssh免密码登录
测试
ssh localhost
必须要输入密码的话执行:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >>~/.ssh/authorized_keys
启动
格式化namenode
bin/hdfsnamenode -format
开启namenode和datanode守护进程
sbin/start-dfs.sh
查看namenode状态通过50070端口
创建hdfs所需的hdfs目录用来执行mapreduce
bin/hdfsdfs -mkdir /user
bin/hdfsdfs -mkdir /user/Hadoop
bin/hdfsdfs -ls /
把input放入hdfs
bin/hdfsdfs -mkdir /input
bin/hdfsdfs -put etc/hadoop /input
跑下例子
bin/hadoopjar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep /inputoutput 'dfs[a-z.]+'
bin/hadoopjar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar grep /input/*output3 'con[a-z.]+'
把结果集从hdfs拿下来
bin/hdfsdfs -get output3 output5
catoutput5/*
停止命令
sbin/stop-dfs.sh
单节点yarn
配置
etc/hadoop/mapred-site.xml:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
etc/hadoop/yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
启动ResourceManager和NodeManager守护进程
sbin/start-yarn.sh
查看ResourceManager的web页面在端口8088
停止yarn
sbin/stop-yarn.sh
完全分布式集群搭建
3台机器
修改hostname
vim/etc/sysconfig/network
修改hostname=yourname
Hostname yourname
修改host
vim/etc/hosts
yourmaster master
yourslaveslave1
yourslaveslave2
scp/etc/hosts root@slave1:/etc/
scp/etc/hosts root@slave2:/etc/
添加用户
useradd -u 600 hadoop
passwd Hadoop
su – hadoop
在Slave1执行命令
ssh-keygen-t dsa -P '' -f ~/.ssh/id_dsa
cat~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
scp /home/hadoop/.ssh/id_dsa.pub.130 hadoop@master:/home/hadoop/.ssh/
# vi /etc/ssh/sshd_config #开启RSA认证
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
#service sshd restart
同理slave2
在master执行
catid_dsa.pub.130 >> authorized_keys
cat id_dsa.pub.131>> authorized_keys
权限很重要
chmod 700 .ssh/
chmod 600 .ssh/authorized_keys
scp/home/hadoop/.ssh/authorized_keys hadoop@slave1:/home/hadoop/.ssh/
scp /home/hadoop/.ssh/authorized_keys hadoop@slave2:/home/hadoop/.ssh/
在master测试
Ssh slave1 ssh slave2
解压hadoop安装包
chown hadoop.hadoop -R/export/servers/hadoop-2.6.0
编辑环境变量
vi /etc/profile
HADOOP_HOME=/home/hadoop/hadoop-2.3.0/
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_HOME PATH
source/etc/profile
编辑slaves
[hadoop@master hadoop]$ vi slaves
slave1
slave2
core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/export/servers/hadoop-2.6.0/tmp</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/export/servers/hadoop-2.6.0/dfs/name</value>
</property>
<property>
<name>dfs.namenode.data.dir</name>
<value>file:/export/servers/hadoop-2.6.0/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value> //数据副本数量 默认为3
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
格式化文件系统并启动
hdfs namenode –format
启动hdfs文件系统
start-dfs.sh
jps
查看守护进程
352NameNode
531SecondaryNameNode
645 Jps
启动mapreduce架构
start-yarn.sh
查看进程
1870ResourceManager
查看slave
sshslave1
jps
26043 NodeManager
查看
http://192.168.159.129:8088/cluster/nodes
可以用浏览器打开NameNode, ResourceManager和各个NodeManager的web界面,
· NameNode web UI, http://master:50070/
· ResourceManager web UI, http://master:8088/
· NodeManager web UI, http://slave01:8042
验证
hdfs dfs-mkdir /user
hdfs dfs-mkdir /user/hadoop
hdfs dfs-put ./etc/hadoop input
运行著名的WordCount
hadoopjar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount inputoutput
查看结果集
hdfs dfs -cat output/*
停止集群
$ stop-yarn.sh
$ stop-hdfs.sh
后记:
因为测试环境机器每周都会重启所以经常面临重启集群的情况:
重启时
hdfs namenode -format
hdfs datanode -format
sbin/start-dfs.sh;tail -f logs/*
sbin/start-yarn.sh;tail -f logs/*
注意查看日志报错