阿里云-hadoop环境搭建

一、 准备环境,实际操作在本地由于电脑性能问题,导致搭建的集群一直进行GC,无法启动成功,所以决定在阿里云服务上进行实践。

 

准备三台云服务器 

系统: centos7.2   vcore 2  ram 8G,购买的是按流量付费,每小时都是钱啊。。。

1、初始化后设置连接密码。

如何连接参考官方文档

本地已经连接OK了

1、 设置初始化环境

  设定host 

 vi  /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
172.31.215.90 hadoop001
172.31.215.89 hadoop002
172.31.215.88 hadoop003

 先关闭防火墙。

[root@izj6c3uhsuxfbdlkkpar59z ~]# systemctl disable firewalld
[root@izj6c3uhsuxfbdlkkpar59z ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@izj6c3uhsuxfbdlkkpar59z ~]# 

关闭selinux 安全策略。

[root@izj6c3uhsuxfbdlkkpar5az ~]# vi /etc/selinux/config 


# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

同步时钟 使用ntp

发现阿里云是安装OK的。

[root@izj6c3uhsuxfbdlkkpar5az ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: disabled)
   Active: active (running) since 六 2019-11-23 09:08:21 CST; 18min ago
 Main PID: 868 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─868 /usr/sbin/ntpd -u ntp:ntp -g

11月 23 09:08:21 izj6c3uhsuxfbdlkkpar5az systemd[1]: Starting Network Time Service...
11月 23 09:08:21 izj6c3uhsuxfbdlkkpar5az systemd[1]: Started Network Time Service.
11月 23 09:08:21 izj6c3uhsuxfbdlkkpar5az ntpd[868]: proto: precision = 0.090 usec
11月 23 09:08:21 izj6c3uhsuxfbdlkkpar5az ntpd[868]: 0.0.0.0 c01d 0d kern kernel time sync enabled
[root@izj6c3uhsuxfbdlkkpar5az ~]# crontab -e

以hadoop001为时间的主节点

[root@izj6c3uhsuxfbdlkkpar5az ~]# vi /etc/ntp.conf 

server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#当外部时间不可用时,可使用本地硬件时间
server 127.127.1.0 iburst local clock
#允许哪些网段的机器来同步时间
restrict 172.31.215.0 mask 255.255.255.0 nomodify notrap

验证

[root@izj6c3uhsuxfbdlkkpar5az ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l  105   64    2    0.000    0.000   0.000
+120.25.115.20   10.137.53.7      2 u   66   64    7   13.823    6.173   4.503
 10.143.33.49    .STEP.          16 u    -  128    0    0.000    0.000   0.000
+100.100.3.1     10.137.55.181    2 u   65   64    7   26.354    6.021   4.514
+100.100.3.2     10.137.55.181    2 u   38   64    3   25.829    2.324   9.414
+100.100.3.3     10.137.55.181    2 u   48   64    7   25.890    3.367   4.482
-203.107.6.88    10.137.38.86     2 u    -   64   17   71.651    2.036  19.031
 10.143.33.50    .STEP.          16 u    -  128    0    0.000    0.000   0.000
 10.143.33.51    .STEP.          16 u    -  128    0    0.000    0.000   0.000
 10.143.0.44     .STEP.          16 u    -  128    0    0.000    0.000   0.000
 10.143.0.45     .STEP.          16 u    -  128    0    0.000    0.000   0.000
 10.143.0.46     .STEP.          16 u    -  128    0    0.000    0.000   0.000
+100.100.5.1     10.137.55.181    2 u   59   64    7   30.301    5.467   4.493
+100.100.5.2     10.137.55.181    2 u   58   64    7   27.384    4.835   4.492
+100.100.5.3     10.137.55.181    2 u   55   64    7   26.931    4.634   4.464
*100.100.61.88   .BD.             1 u   33   64    3    0.181    1.606   9.699

关闭hadoop002 、 hadoop003  ntpd 服务,使他们从hadooop001进行时间同步

[root@izj6c3uhsuxfbdlkkpar59z ~]# systemctl stop ntpd
[root@izj6c3uhsuxfbdlkkpar59z ~]# systemclt disable ntpd
-bash: systemclt: 未找到命令
[root@izj6c3uhsuxfbdlkkpar59z ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@izj6c3uhsuxfbdlkkpar59z ~]# systemctl status ntpd
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

11月 23 08:53:08 izj6c3uhsuxfbdlkkpar59z systemd[1]: Starting Network Time Service...
11月 23 08:53:08 izj6c3uhsuxfbdlkkpar59z systemd[1]: Started Network Time Service.
11月 23 08:53:08 izj6c3uhsuxfbdlkkpar59z ntpd[828]: proto: precision = 0.072 usec
11月 23 08:53:08 izj6c3uhsuxfbdlkkpar59z ntpd[828]: 0.0.0.0 c01d 0d kern kernel time sync enabled
11月 23 09:33:45 hadoop002 systemd[1]: Stopping Network Time Service...
11月 23 09:33:45 hadoop002 systemd[1]: Stopped Network Time Service.

同步时间从hadoop0001

问题:

[root@izj6c3uhsuxfbdlkkpar59z ~]# /usr/sbin/ntpdate hadoop001
23 Nov 09:38:45 ntpdate[8022]: no server suitable for synchronization found

防火墙已经关闭了,经排查需要更新ntp即可,

[root@izj6c3uhsuxfbdlkkpar5bz ~]# yum install -y ntp
已加载插件:fastestmirror
base                                                                                                                                   | 3.6 kB  00:00:00     
epel                                                                                                                                   | 5.3 kB  00:00:00     
extras                                                                                                                                 | 2.9 kB  00:00:00     
updates                                                                                                                                | 2.9 kB  00:00:00     
(1/7): base/7/x86_64/group_gz                                                                                                          | 165 kB  00:00:00     
(2/7): epel/x86_64/group_gz                                                                                                            |  90 kB  00:00:00     
(3/7): epel/x86_64/updateinfo                                                                                                          | 1.0 MB  00:00:00     
(4/7): extras/7/x86_64/primary_db                                                                                                      | 153 kB  00:00:00     
(5/7): base/7/x86_64/primary_db                                                                                                        | 6.0 MB  00:00:00     
(6/7): epel/x86_64/primary_db                                                                                                          | 6.9 MB  00:00:00     
(7/7): updates/7/x86_64/primary_db                                                                                                     | 4.2 MB  00:00:00     
Determining fastest mirrors
正在解决依赖关系
--> 正在检查事务
---> 软件包 ntp.x86_64.0.4.2.6p5-25.el7.centos.2 将被 升级
---> 软件包 ntp.x86_64.0.4.2.6p5-29.el7.centos 将被 更新
--> 正在处理依赖关系 ntpdate = 4.2.6p5-29.el7.centos,它被软件包 ntp-4.2.6p5-29.el7.centos.x86_64 需要
--> 正在处理依赖关系 libcrypto.so.10(OPENSSL_1.0.2)(64bit),它被软件包 ntp-4.2.6p5-29.el7.centos.x86_64 需要
--> 正在检查事务
---> 软件包 ntpdate.x86_64.0.4.2.6p5-25.el7.centos.2 将被 升级
---> 软件包 ntpdate.x86_64.0.4.2.6p5-29.el7.centos 将被 更新
---> 软件包 openssl-libs.x86_64.1.1.0.1e-60.el7_3.1 将被 升级
--> 正在处理依赖关系 openssl-libs(x86-64) = 1:1.0.1e-60.el7_3.1,它被软件包 1:openssl-1.0.1e-60.el7_3.1.x86_64 需要
---> 软件包 openssl-libs.x86_64.1.1.0.2k-19.el7 将被 更新
--> 正在检查事务
---> 软件包 openssl.x86_64.1.1.0.1e-60.el7_3.1 将被 升级
---> 软件包 openssl.x86_64.1.1.0.2k-19.el7 将被 更新
--> 解决依赖关系完成

依赖关系解决

==============================================================================================================================================================
 Package                                架构                             版本                                            源                              大小
==============================================================================================================================================================
正在更新:
 ntp                                    x86_64                           4.2.6p5-29.el7.centos                           base                           548 k
为依赖而更新:
 ntpdate                                x86_64                           4.2.6p5-29.el7.centos                           base                            86 k
 openssl                                x86_64                           1:1.0.2k-19.el7                                 base                           493 k
 openssl-libs                           x86_64                           1:1.0.2k-19.el7                                 base                           1.2 M

事务概要
==============================================================================================================================================================
升级  1 软件包 (+3 依赖软件包)

总下载量:2.3 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/4): ntpdate-4.2.6p5-29.el7.centos.x86_64.rpm                                                                                        |  86 kB  00:00:00     
(2/4): ntp-4.2.6p5-29.el7.centos.x86_64.rpm                                                                                            | 548 kB  00:00:00     
(3/4): openssl-1.0.2k-19.el7.x86_64.rpm                                                                                                | 493 kB  00:00:00     
(4/4): openssl-libs-1.0.2k-19.el7.x86_64.rpm                                                                                           | 1.2 MB  00:00:00     
--------------------------------------------------------------------------------------------------------------------------------------------------------------
总计                                                                                                                           19 MB/s | 2.3 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  正在更新    : 1:openssl-libs-1.0.2k-19.el7.x86_64                                                                                                       1/8 
  正在更新    : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                      2/8 
  正在更新    : ntp-4.2.6p5-29.el7.centos.x86_64                                                                                                          3/8 
  正在更新    : 1:openssl-1.0.2k-19.el7.x86_64                                                                                                            4/8 
  清理        : ntp-4.2.6p5-25.el7.centos.2.x86_64                                                                                                        5/8 
  清理        : ntpdate-4.2.6p5-25.el7.centos.2.x86_64                                                                                                    6/8 
  清理        : 1:openssl-1.0.1e-60.el7_3.1.x86_64                                                                                                        7/8 
  清理        : 1:openssl-libs-1.0.1e-60.el7_3.1.x86_64                                                                                                   8/8 
  验证中      : ntp-4.2.6p5-29.el7.centos.x86_64                                                                                                          1/8 
  验证中      : ntpdate-4.2.6p5-29.el7.centos.x86_64                                                                                                      2/8 
  验证中      : 1:openssl-1.0.2k-19.el7.x86_64                                                                                                            3/8 
  验证中      : 1:openssl-libs-1.0.2k-19.el7.x86_64                                                                                                       4/8 
  验证中      : 1:openssl-1.0.1e-60.el7_3.1.x86_64                                                                                                        5/8 
  验证中      : 1:openssl-libs-1.0.1e-60.el7_3.1.x86_64                                                                                                   6/8 
  验证中      : ntp-4.2.6p5-25.el7.centos.2.x86_64                                                                                                        7/8 
  验证中      : ntpdate-4.2.6p5-25.el7.centos.2.x86_64                                                                                                    8/8 

更新完毕:
  ntp.x86_64 0:4.2.6p5-29.el7.centos                                                                                                                          

作为依赖被升级:
  ntpdate.x86_64 0:4.2.6p5-29.el7.centos                  openssl.x86_64 1:1.0.2k-19.el7                  openssl-libs.x86_64 1:1.0.2k-19.el7                 

完毕!
[root@izj6c3uhsuxfbdlkkpar5bz ~]# ntpdate hadoop001
23 Nov 09:47:54 ntpdate[10977]: adjust time server 172.31.215.90 offset 0.118699 sec

设置 crontab 进行周期同步时间。

[root@izj6c3uhsuxfbdlkkpar5bz ~]# crontab -e

00 00 * * * ntpdate hadoop001

二、 上传资源包

 mkdir /home/cdh5.16

 

  • 1
    点赞
  • 2
    收藏
    觉得还不错? 一键收藏
  • 0
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值