Ambari安装与集群配置

最新推荐文章于 2024-05-10 02:19:37 发布

程裕强

最新推荐文章于 2024-05-10 02:19:37 发布

阅读量6.4k

点赞数 1

分类专栏：大数据 Ambari运维文章标签： ambari ambari-server

本文链接：https://blog.csdn.net/chengyuqiang/article/details/53997009

版权

大数据同时被 2 个专栏收录

28 篇文章 0 订阅

订阅专栏

Ambari运维

11 篇文章 1 订阅

订阅专栏

参考网址：
http://www.ibm.com/developerworks/cn/opensource/os-cn-bigdata-ambari/index.html
http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/index.html

第一部分：基本环境准备

   准备3个节点，下面操作需要在每个节点上执行。
   参考网址http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/prepare_the_environment.html

1、配置IP和hostname
   [root@node1 ~]# vi /etc/sysconfig/network-scripts/ifcfg-eno16777736
    [root@node1 ~]# vi /etc/hosts
   注意：不要从您的hosts文件中删除以下两行。删除或编辑以下行可能会导致需要网络功能的各种程序失败。
   127.0.0.1 localhost.localdomain localhost
   ::1 localhost6.localdomain6 localhost6
    [root@node1 ~]# vi /etc/hostname
   [root@node1 ~]# service network restart

2、更改yum源
2.1 下载国内镜像源repo文件
   如果已经安装wget工具，则可以通过下面命令下，自动载配置
   [root@node1 ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
   --2017-01-02 21:44:57-- http://mirrors.aliyun.com/repo/Centos-7.repo
   Resolving mirrors.aliyun.com (mirrors.aliyun.com)... 115.28.122.210, 112.124.140.210
   Connecting to mirrors.aliyun.com (mirrors.aliyun.com)|115.28.122.210|:80... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 2573 (2.5K) [application/octet-stream]
   Saving to: ‘/etc/yum.repos.d/CentOS-Base.repo’
   100%[========================================================================>] 2,573       --.-K/s   in 0s
   2017-01-02 21:44:58 (231 MB/s) - ‘/etc/yum.repos.d/CentOS-Base.repo’ saved [2573/2573]
    如果没有安装wget工具，直接下载http://mirrors.aliyun.com/repo/Centos-7.repo
   然后将下载的阿里镜像源文件移到/etc/yum.repos.d下即可（重命名CentOS-Base.repo）
   [root@node1 ~]# mv Centos-7.repo /etc/yum.repos.d/CentOS-Base.repo
2.2 更新yum缓存
   [root@node1 ~]# yum clean all
   Loaded plugins: fastestmirror
   Cleaning repos: base c7-media extras updates
   Cleaning up everything
   Cleaning up list of fastest mirrors
   [root@node1 ~]# yum makecache
   ......
   Determining fastest mirrors
   * base: mirrors.aliyun.com
   * extras: mirrors.aliyun.com
   * updates: mirrors.aliyun.com
   Metadata Cache Created

3、关闭防火墙和SELINUX
   [root@node1 ~]# systemctl disable firewalld
     Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
     Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
    [root@node1 ~]# service firewalld stop
     Redirecting to /bin/systemctl stop firewalld.service
    [root@node1 ~]# setenforce 0
    [root@node1 ~]# vi /etc/selinux/config
     SELINUX=disabled
   设置umask值
   [root@node1 ~]echo umask 0022 >> /etc/profile

4、设置SSH无密钥登录

4.1 每个节点生成无密钥SSH

[root@node1 ~]#yum install openssh-server

    [root@node1 ~]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
    [root@node1 ~]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
    [root@node1 ~]# chmod 0600 ~/.ssh/authorized_keys

4.2   相互访问
   [root@anode1 ~]#vi /etc/hosts
   127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
   ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
   172.16.54.201 anode1
   172.16.54.202 anode2
   172.16.54.203 anode3
   [root@anode1 ~]#scp /etc/hosts 172.16.54.202:/etc/

[root@anode1 ~]#scp /etc/hosts 172.16.54.203:/etc/

然后在每个节点执行sh sshKeyGen.sh

5、设置时间同步
   [root@node1 ~]# yum install -y ntp
    [root@node1 ~]# systemctl enable ntpd
     Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
    [root@node1 ~]# systemctl start ntpd

6、安装配置JDK 1.8
6.1 卸载预装的JDK
   rpm -qa|grep jdk
   yum -y remove java java-1.7.0-openjdk-1.7.0.79-2.5.5.4.el6.x86_64
   rpm -qa|grep java
   yum -y remove java tzdata-java-2015e-1.el6.noarch
6.2 配置新的JDK
   注意，每个节点的jdk安装到相同的目录下面，比如都安装到/opt目录下。
   [root@node1 ~]#vi /etc/profile
   #set Java Path
   export JAVA_HOME=/opt/java/jdk1.8.0_101
   export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
   export PATH=$PATH:$JAVA_HOME/bin
   [root@node1 ~]# source /etc/profile
   [root@node1 ~]# java -version

7、拾遗

7.1 设置buildNumber

这一部分是针对于ambari源码编译出现buildNumber缺失一种解决办法。

   [root@node1 ~]# vi .bash_profile
   export buildNumber=9999
   [root@anode1 ~]# source .bash_profile
   [root@node1 ~]# echo $buildNumber
   9999
7.2 确认是否安装了wget
   [root@anode1 ~]# yum install -y wget


第二部分 Ambari-server安装与配置

1、下载Ambari 的公共库文件（public repository）
   http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo
   [root@anode2 ~]# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
   2017-01-03 06:41:08 URL:http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari.repo [287/287] -> "/etc/yum.repos.d/ambari.repo" [1]

2、修改ambari.repo文件
   如果访问hortonworks.com速度顺畅，则不要修改ambari.repo，可以跳过这一步。
   当访问hortonworks.com速度较慢时，可以建立本地repository。
   [root@vnode0 yum.repos.d]# cat ambari.repo
   #VERSION_NUMBER=2.4.2.0-136
   [Updates-ambari-2.4.2.0]
   name=ambari-2.4.2.0 - Updates
   baseurl=http://192.168.1.180/ambari/centos7/2.4.2.0-136/
   gpgcheck=0
   gpgkey=http://192.168.1.180/ambari/centos7/2.4.2.0-136//RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
   enabled=1

priority=1

然后将ambari.repo复制到其他节点
[root@node1 yum.repos.d]# scp ambari.repo 172.16.54.202:/etc/yum.repos.d/

3、安装Ambari-server
   [root@east81 yum.repos.d]# yum install ambari-server
   Loaded plugins: fastestmirror, langpacks
   Loading mirror speeds from cached hostfile
   Resolving Dependencies

   ==================================================================================================================
   Package                                    Arch                            Version                                    Repository                                       Size
   =====================================================================================================================
   Installing:
   ambari-server                              x86_64                          2.4.2.0-136                                Updates-ambari-2.4.2.0                          645 M
   Installing for dependencies:
   postgresql                                 x86_64                          9.2.13-1.el7_1                             os                                              3.0 M
   postgresql-libs                            x86_64                          9.2.13-1.el7_1                             os                                              230 k
   postgresql-server                          x86_64                          9.2.13-1.el7_1                             os                                              3.8 M

   Transaction Summary
   ======================================================================================================================
   Install 1 Package (+3 Dependent packages)

   Total download size: 652 M
   Installed size: 732 M
   Is this ok [y/d/N]: y
   Downloading packages:
   (1/4): postgresql-libs-9.2.13-1.el7_1.x86_64.rpm                                                                                                      | 230 kB 00:00:00
   (2/4): postgresql-9.2.13-1.el7_1.x86_64.rpm                                                                                                           | 3.0 MB 00:00:00
   (3/4): postgresql-server-9.2.13-1.el7_1.x86_64.rpm                                                                                                    | 3.8 MB 00:00:00
   ambari-server-2.4.2.0-136.x86_ FAILED                                                                                                     ] 159 B/s | 7.8 MB 1180:43:26 ETA
   http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari/ambari-server-2.4.2.0-136.x86_64.rpm: [Errno 12] Timeout on http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari/ambari-server-2.4.2.0-136.x86_64.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
   Trying other mirror.
   ambari-server-2.4.2.0-136.x86_ FAILED                                          -                                                           ] 437 B/s | 31 MB 414:04:07 ETA
   http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari/ambari-server-2.4.2.0-136.x86_64.rpm: [Errno 12] Timeout on http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.2.0/ambari/ambari-server-2.4.2.0-136.x86_64.rpm: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
   Trying other mirror.
   warning: /var/cache/yum/x86_64/7/Updates-ambari-2.4.2.0/packages/ambari-server-2.4.2.0-136.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 07513cad: NOKEY MB 00:00:00 ETA
   Public key for ambari-server-2.4.2.0-136.x86_64.rpm is not installed
   (4/4): ambari-server-2.4.2.0-136.x86_64.rpm                                                                                                           | 645 MB 00:17:12
   -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   Total                                                                                                                                        295 kB/s | 652 MB 00:37:44
   Retrieving key from http://public-repo-1.hortonworks.com/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
   Importing GPG key 0x07513CAD:
   Userid     : "Jenkins (HDP Builds) <jenkin@hortonworks.com>"
   Fingerprint: df52 ed4f 7a3a 5882 c099 4c66 b973 3a7a 0751 3cad
   From       : http://public-repo-1.hortonworks.com/ambari/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
   Is this ok [y/N]: y
   Running transaction check
   Running transaction test
   Transaction test succeeded
   Running transaction
      Installing : postgresql-libs-9.2.13-1.el7_1.x86_64                                                                                                           1/4
      Installing : postgresql-9.2.13-1.el7_1.x86_64                                                                                                                2/4
      Installing : postgresql-server-9.2.13-1.el7_1.x86_64                                                                                                         3/4
      Installing : ambari-server-2.4.2.0-136.x86_64                                                                                                                4/4
      Verifying : postgresql-9.2.13-1.el7_1.x86_64                                                                                                                1/4
      Verifying : ambari-server-2.4.2.0-136.x86_64                                                                                                                2/4
      Verifying : postgresql-libs-9.2.13-1.el7_1.x86_64                                                                                                           3/4
      Verifying : postgresql-server-9.2.13-1.el7_1.x86_64                                                                                                         4/4
   Installed:
      ambari-server.x86_64 0:2.4.2.0-136
   Dependency Installed:
      postgresql.x86_64 0:9.2.13-1.el7_1              postgresql-libs.x86_64 0:9.2.13-1.el7_1               postgresql-server.x86_64 0:9.2.13-1.el7_1
   Complete!
   [root@east81 yum.repos.d]#
4、设置Ambari-server
   [root@anode1 opt]# ambari-server setup
   Using python /usr/bin/python
   Setup ambari-server
   Checking SELinux...
   SELinux status is 'disabled'
   Customize user account for ambari-server daemon [y/n] (n)? y    #输入y
   Enter user account for ambari-server daemon (root):             #回车，默认值
   Adjusting ambari-server permissions and ownership...
   Checking firewall status...
   Checking JDK...
   [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
   [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
   [3] Custom JDK
   ==============================================================================
   Enter choice (1): 3                                             #输入3
   WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
   WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
   Path to JAVA_HOME: /opt/jdk1.8.0_101                            #输入JDK根目录
   Validating JDK on Ambari Server...done.
   Completing setup...
   Configuring database...
   Enter advanced database configuration [y/n] (n)?                #回车，默认PostgreSQL
   Configuring database...
   Default properties detected. Using built-in database.
   Configuring ambari database...
   Checking PostgreSQL...
   Running initdb: This may take up to a minute.
   Initializing database ... OK
   About to start PostgreSQL
   Configuring local database...
   Connecting to local database...done.
   Configuring PostgreSQL...
   Restarting PostgreSQL
   Extracting system views...
   ambari-admin-2.4.2.0.0.jar
   Adjusting ambari-server permissions and ownership...
   Ambari Server 'setup' completed successfully.
   [root@anode1 opt]#

5、启动Ambari-server
   [root@anode1 opt]# ambari-server start
   Using python /usr/bin/python
   Starting ambari-server
   Ambari Server running with administrator privileges.
   Organizing resource files at /var/lib/ambari-server/resources...
   Ambari database consistency check started...
   No errors were found.
   Ambari database consistency check finished
   Server PID at: /var/run/ambari-server/ambari-server.pid
   Server out at: /var/log/ambari-server/ambari-server.out
   Server log at: /var/log/ambari-server/ambari-server.log
   Waiting for server start....................
   Ambari Server 'start' completed successfully.

第三部分集群配置

1、登录Ambari-server
    比如，Ambari-server所在节点node1的IP是172.16.54.201
   在浏览器中打开http://172.16.54.201:8080
   用户名和密码都是admin,即可登录Ambari-server
2、单击“Launch Install Wizard”按钮，启动Ambari安装向导
    (1) 在“Name your cluster”输入框输入集群名称，比如cc
   (2) Select Version选择版本,选择HDP2.4
       如果“使用公共存储库”，保留“redhat7”即可，其他系统删除。
       如果“使用本地存储库”，同样保留“redhat7”即可，删除其他系统。修改：
        HDP-2.4输入：http://192.168.1.180/hdp/HDP/centos7/2.x/updates/2.4.3.0/

HDP-UTILS-1.1.0.20输入：http://192.168.1.180/hdp/HDP-UTILS-1.1.0.20/repos/centos7/

提示：会在各个节点生成hdp repo文件

[root@a1 yum.repos.d]# ls
ambari.repo bak CentOS-Base.repo HDP.repo HDP-UTILS.repo

       [root@a1 yum.repos.d]# more HDP.repo
       [HDP-2.4]
       name=HDP-2.4
       baseurl=http://192.168.1.60/hdp/HDP/centos7/2.x/updates/2.4.3.0/
       path=/
       enabled=1
       gpgcheck=0
       [root@a1 yum.repos.d]# more HDP-UTILS.repo
       [HDP-UTILS-1.1.0.20]
       name=HDP-UTILS-1.1.0.20
       baseurl=http://192.168.1.60/hdp/HDP-UTILS-1.1.0.20/repos/centos7/
       path=/
       enabled=1
       gpgcheck=0

    (3)Install Options安装选项
       在Target Hosts：输入节点的hostname值，不是IP。比如：
       anode1
       anode2
       anode3
   (4) Host Registration Information主机注册信息
       将Ambari-server主机的SSH私钥复制到输入框
       [root@anode1 ~]# cat /root/.ssh/id_rsa
   (5) 确认主机，进行anode1、anode2、anode3节点注册
   (6)   选择服务,先选择最小服务安装,后续再安装其他服务。
        建议选择ZooKeeper和Ambari Metrics
   (7)设置Grafana Admin Password
       输入123456即可
   (8)分配从属机和客户端,所有节点选择安装Client


第四部分：遇到问题

1、警告 Ambari Metrics    hbase_master_heapsize    512
   Value is less than the recommended default of 640
   HBase Master Heap Size. In embedded mode, total heap size is sum of master and regionserver heap sizes.
   解决办法：
   Ambari Metrics配置，检索hbase_master_heapsize，然后将值修改为640，单击保存按钮
   最后重启组件。

2、ambari 报Heartbeat Lost问题

重启ambari-agent

[root@anode1 ~]# ambari-server start