##安装hadoop集群:
步骤如下:
- 通过xmanager的Xftp上传hadoop-2.6.4.tar.gz文件到/opt目录
- 备注#没有xmanager的可以去下载哦,该软件很好用
- 解压缩hadoop-2.6.4.tar.gz 文件,进入该目录tar -zxf hadoop-2.6.4.tar.gz -C /usr/local解压后即可,看到/usr/local/hadoop-2.6.4文件夹
- mv hadoop-2.6.4 Hadoop此时我已经更改hadoop-2.6.4为Hadoop,以后注意得把hadoop-2.6.4改为Hadoop,当然,你也可以选择不改,但是需要保持同样即可(和你的一致哦);
- 配置Hadoop进入目录:cd /usr/local/hadoop/etc/hadoop/依次修改下面的文件
- :4.1 core-site.xml 文件 fs.defaultFS hdfs://master:8020 hadoop.tmp.dir /var/log/hadoop/tmp 4.2 hadoop-env.sh
- 备注:javahome选择你的路径哦,echo $JAVA_HOME即可知道你的路径
- export JAVA_HOME=/usr/local/jdk1.8.0_171
- 4.3 hdfs-site.xml文件 dfs.namenode.name.dir file:///data/hadoop/hdfs/name dfs.datanode.data.dir file:///data/hadoop/hdfs/data dfs.namenode.secondary.http-address master:50090 dfs.replication 3
- 4.4 mapred-site.xml文件
- 备注:该目录只有mapred-site.xml.template文件,没有mapred-site.xml,所以需要改名或者说复制
- 复制cp mapred-site.xml.template mapred-site.xml
- mapreduce.framework.name yarn mapreduce.jobhistory.address master:10020 mapreduce.jobhistory.webapp.address master:19888
- 4.5 yarn-site.xml 文件
- yarn.resourcemanager.hostname master yarn.resourcemanager.address y a r n . r e s o u r c e m a n a g e r . h o s t n a m e : 8032 < / v a l u e > < / p r o p e r t y > < p r o p e r t y > < n a m e > y a r n . r e s o u r c e m a n a g e r . s c h e d u l e r . a d d r e s s < / n a m e > < v a l u e > {yarn.resourcemanager.hostname}:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value> yarn.resourcemanager.hostname:8032</value></property><property><name>yarn.resourcemanager.scheduler.address</name><value>{yarn.resourcemanager.hostname}:8030 yarn.resourcemanager.webapp.address y a r n . r e s o u r c e m a n a g e r . h o s t n a m e : 8088 < / v a l u e > < / p r o p e r t y > < p r o p e r t y > < n a m e > y a r n . r e s o u r c e m a n a g e r . w e b a p p . h t t p s . a d d r e s s < / n a m e > < v a l u e > {yarn.resourcemanager.hostname}:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.https.address</name> <value> yarn.resourcemanager.hostname:8088</value></property><property><name>yarn.resourcemanager.webapp.https.address</name><value>{yarn.resourcemanager.hostname}:8090 yarn.resourcemanager.resource-tracker.address y a r n . r e s o u r c e m a n a g e r . h o s t n a m e : 8031 < / v a l u e > < / p r o p e r t y > < p r o p e r t y > < n a m e > y a r n . r e s o u r c e m a n a g e r . a d m i n . a d d r e s s < / n a m e > < v a l u e > {yarn.resourcemanager.hostname}:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value> yarn.resourcemanager.hostname:8031</value></property><property><name>yarn.resourcemanager.admin.address</name><value>{yarn.resourcemanager.hostname}:8033 yarn.nodemanager.local-dirs /data/hadoop/yarn/local yarn.log-aggregation-enable true yarn.nodemanager.remote-app-log-dir /data/tmp/logs yarn.log.server.url http://master:19888/jobhistory/logs/ URL for job history server yarn.nodemanager.vmem-check-enabled false yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.nodemanager.resource.memory-mb 2048 yarn.scheduler.minimum-allocation-mb 512 yarn.scheduler.maximum-allocation-mb 4096 mapreduce.map.memory.mb 2048 mapreduce.reduce.memory.mb 2048 yarn.nodemanager.resource.cpu-vcores 1
- 4.6 yarn-env.sh文件
- 备注:注意Javahome位置路径
- export JAVA_HOME=/usr/local/jdk1.8.0_171
- 4.7 slaves文件
- 备注:slave1,slave2,slave3为我的三个子节点,你选择自身的情况内容改为一下:
- slave1
- slave2
- slave3
- 4.8 设置IP映射编辑各节点/etc/hosts
- 备注:192.168.128.130为主节点master的IP地址,其他子节点slave的IP地址也是相对应的,要根据自身情况
- 添加:
- 192.168.128.130 master master.centos.com
- 192.168.128.131 slave1 slave1.centos.com
- 192.168.128.132 slave2 slave2.centos.com
- 192.168.128.133 slave3 slave3.centos.com
- 拷贝hadoop安装文件到集群slave节点
- scp -r /usr/local/hadoop slave1:/usr/local
- scp -r /usr/local/hadoop slave2:/usr/local
- scp -r /usr/local/hadoop slave3:/usr/local
- 5.配置SSH无密码登录
- (1)使用ssh-keygen产生公钥与私钥对。输入命令“ssh-keygen -t rsa”,接着按三次Enter键
- [root@master ~]# ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id_rsa): Created directory ‘/root/.ssh’.Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa.Your public key has been saved in /root/.ssh/id_rsa.pub.The key fingerprint is:a6:13:5a:7b:54:eb:77:58:bd:56:ef:d0:64:90:66:d4 root@master.centos.comThe key’s randomart image is:±-[ RSA 2048]----+| … || . .E|| . = || . . o o || o S . . =|| o * . o ++|| . + . . o ooo|| o . …o || .|±----------------+生成私有密钥id_rsa和公有密钥id_rsa.pub两个文件。ssh-keygen用来生成RSA类型的密钥以及管理该密钥,参数“-t”用于指定要创建的SSH密钥的类型为RSA。
- (2)用ssh-copy-id将公钥复制到远程机器中 ##该步骤的master,slave等子节点用IP地址永远不会错。
- ssh-copy-id -i /root/.ssh/id_rsa.pub master //依次输入yes,123456(root用户的密码),也就是你进入xshell的密码
- ssh-copy-id -i /root/.ssh/id_rsa.pub slave1
- ssh-copy-id -i /root/.ssh/id_rsa.pub slave2
- ssh-copy-id -i /root/.ssh/id_rsa.pub slave3
- (3)验证是否设置无密码登录依次输入
- ssh slave1
- ssh slave2
- ssh slave3
- 6.配置时间同步服务
- (1)安装NTP服务。
- 在各节点:yum -y install ntp
- (2)设置假设master节点为NTP服务主节点,那么其配置如下。
- 使用命令“vim /etc/ntp.conf”打开/etc/ntp.conf文件,注释掉以server开头的行,并添加:
- restrict 192.168.0.0 mask 255.255.255.0 nomodify notrap
- server 127.127.1.0
- fudge 127.127.1.0 stratum 10
- (3)在slave中配置NTP,同样修改/etc/ntp.conf文件,注释掉server开头的行,并添加:
- server master
- (4)执行命令“service iptables stop & chkconfig iptables off”永久性关闭防火墙,主节点和从节点都要关闭。
- (5)启动NTP服务。
- ① 在master节点执行命令“service ntpd start & chkconfig ntpd on”
- ② 在slave上执行命令“ntpdate master”即可同步时间
- ③ 在slave上分别执行“service ntpd start & chkconfig ntpd on”即可启动并永久启动NTP服务。
- 7.在/etc/profile添加JAVA_HOME和Hadoop路径 ##记得,这些路径都是要根据自身情况来定,不要只顾着Ctrl+c和Ctrl+v哈
- export HADOOP_HOME=/usr/local/hadoop
- export HADOOP_HOME=/usr/local/hadoop
- export JAVA_HOME=/usr/local/jdk1.8.0_171
- export PATH= H A D O O P H O M E / b i n : HADOOP_HOME/bin: HADOOPHOME/bin:PATH:$JAVA_HOME/bin
- source /etc/profile使修改生效
备注:该命令常用,以后加入其他路径也要用到,建议记住哦
- 格式化NameNode进入目录cd /usr/local/hadoop/bincd /usr/local/hadoop/bin
- 执行格式化 ./hdfs namenode -format
- 启动集群进入目录cd /usr/local/hadoop/sbincd /usr/local/hadoop/sbin执行启动:
- ./start-dfs.sh./start-yarn.sh./mr-jobhistory-daemon.sh start historyserver
- 使用jps,查看进程 备注:前面数字是随机的,不一样没关系,但是后面的要有哦,不然就说明你的有错误了,要回去检查一下自身的命令以及输入的内容哦
- [root@centos67 sbin]# jps3672 NodeManager3301 DataNode3038 NameNode4000 JobHistoryServer4058 Jps3589 ResourceManager3408 SecondaryNameNode
-
- 关闭防火墙(在所有节点执行):
service iptables stop chkconfig iptables off
浏览器查看: ##如果不行,那就先去进行第九步哦,在Windows上进行,步骤和Linux上的一模一样哦。192.168.128.130:50070/dfshealth.jsp
- 关闭防火墙(在所有节点执行):
http://master:50070http://master:80889.
在Windows系统上,设置IP映射(和Linux一样的操作)更改:C:\Windows\System32\drivers\etc\hosts文件 ###直接不可编辑,可以直接拉出桌面进行编辑。添加如下内容:
192.168.128.130 master master.centos.com
192.168.128.131 slave1 slave1.centos.com
192.168.128.132 slave2 slave2.centos.com
192.168.128.133 slave3 slave3.centos.com