Ambari的安装

1.centos 7.5的安装
    安装了三台机器（一台bin、两台minimal） 
    -----------------------------------------------------------
    节点名称                系统版本                    内存        磁盘        
    zte.hdp.bigdata1    centos7.5/bin            6G        100G    
    zte.hdp.bigdata2    centos7.5/minimal        2G        20G
    zte.hdp.bigdata3    centos7.5/minimal        2G        20G


    1）与centos6.x不同的是，网络文件是/etc/sysconfig/network-scripts/ifcfg-ens33
        
        除了BOOTPRTO=static和ONBOOT=yes ，还要添加：
        
        # jwb profile
        GATEWAY=192.168.159.2
        IPADDR=192.168.159.11
        NETMASK=255.255.255.0
        DNS1=8.8.8.8
        DNS2=8.8.4.4

        然后重新启动网卡：
            systemctl restart network  (重启网络服务)
            systemctl start network   （启动网络服务）
            systemctl stop network    （停止网络服务）

    2）三台都安装完毕后，配置映射，并在PC配置映射，然后通过xshell6（破解版超好用）ssh连接

        vim /etc/hosts
            # HDP大数据集群
            192.168.159.11 zte.hdp.bigdata1 zte1
            192.168.159.12 zte.hdp.bigdata2 zte2
            192.168.159.13 zte.hdp.bigdata3 zte3

2.配置免密登录
    ssh-keygen -t rsa    （三台机器都执行）
    ssh-copy-id zte.hdp.bigdata1

    scp /root/.ssh/authorized_keys zte.hdp.bigdata2:/root/.ssh
    scp /root/.ssh/authorized_keys zte.hdp.bigdata3:/root/.ssh

    注意：zte02和zte03是minimal版本，很多库都没有，要自己安装
        yum install -y vim 

3.基础配置
    1）禁用selinux
        vim /etc/selinux/config
            SELINUX=disabled

    2)分别关闭防火墙
        systemctl disable firewalld
        systemctl stop firewalld

        说明:两个命令都要敲，systemctl是开机启动关闭，stop 是立刻关闭

    3)分别修改linux对资源的限制
        vim /etc/security/limits.conf

            *    soft    nofile    100001
            *    hard    nofile    100002
            root    soft    nofile    100001
            root    hard    nofile    100002

        然后scp给另外两台

    4）集群中的时钟必须同步
        方案1:所有集群机器都和阿里云等中央时钟同步
        方案2：其他机器和其中一台同步

        现在进行局域网内容的NTP同步配置：局域网内1台作为NTP Server ，其他服务器作为NTP Client与服务器进行时钟同步

        1>> 检查ntp包是否已经安装，没有则需要安装
            rpm -qa | grep ntp
            yum -y install ntp


        2>> 修改NTP Server配置文件: 
            vim /etc/ntp.conf
                # 新增：日志目录
                logfile /var/log/ntpd.log
                # 新增：授权192.168.159.0这个网段上所有机器可以从这台机器上查询和同步时间
                restrict 192.168.159.0 mask 255.255.255.0 nomodify notrap
                # 新增：时间服务器（将原来的注释掉）
                server 0.cn.pool.ntp.org iburst
                server 1.cn.pool.ntp.org iburst 
                server 2.cn.pool.ntp.org iburst
                server 3.cn.pool.ntp.org iburst
                # 新增：外部事件不可用时，使用本地时间
                server 192.168.139.11 iburst
                fudge 127.0.0.1 stratum 10 
                # 新增：允许上层时间服务器主动修改本机时间
                restrict 0.cn.pool.ntp.org nomodify notrap nopeer noquery
                restrict 1.cn.pool.ntp.org nomodify notrap nopeer noquery
                restrict 2.cn.pool.ntp.org nomodify notrap nopeer noquery
            
            设置系统开机启动：
                systemctl enable ntpd
                systemctl enable ntpdate
                systemctl is-enabled ntpd

            校准NTP Server时间：
                ntpdate -u 1.cn.pool.ntp.org

            启动NTP Server：
                systemctl start ntpdate
                systemctl start ntpd


            加入防火墙：
                firewall-cmd --permanent --add-service=ntp
                firewall-cmd --reload

            查看ntp连接状态，没问题则写到硬件
                ss -tlunp | grep ntp
                ntpq -p
                hwclock -w
        


        3>> 修改NTP Client配置文件:
            vim /etc/ntp.conf
            server 192.168.159.11

            重启服务
                systemctl restart ntpd
            以crontab任务计划同步时间（需安装ntpdate，每天24点准时更新同步时间）
                # crontab -e
                0 0 * * * /usr/sbin/sntp -P no -r 192.168.159.11; hwclock -w


    5）安装HTTP服务
        yum -y install httpd
        
        启动服务
            systemctl start httpd

        设置开机启动
            systemctl enable httpd.service

        其他命令
            systemctl start httpd.service #启动
            systemctl stop httpd.service #停止
            systemctl restart httpd.service #重启
            第二、设置开机启动/关闭
            systemctl enable httpd.service #开机启动
            systemctl disable httpd.service #开机不启动
            第三、检查httpd状态
            systemctl status httpd.service





4.基本软件的安装


    统一新建文件夹
        mkdir -p /opt/softs
        mkdir -p /opt/modules
        mkdir -p /opt/shells
        mkdir -p /opt/files
        mkdir -p /opt/temps
        mkdir -p /opt/logs
        mkdir -p /opt/jars
        mkdir -p /opt/exec
        mkdir -p /opt/utils


    1）jdk1.8的按照与配置
        （1）查看有没有已经存在
                rpm -qa | grep java
            有的话就卸载
                rpm -e xxx xxxx xxxxxx --nodeps

            （主要是bin版本的系统需要卸载，minimal本身没有openJdk的）


        （2）上传后解压，然后配置

            [root@zte jdk1.8.0_131]# rpm -qa | grep java
            java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64
            tzdata-java-2018c-1.el7.noarch
            python-javapackages-3.4.1-11.el7.noarch
            javapackages-tools-3.4.1-11.el7.noarch
            java-1.8.0-openjdk-headless-1.8.0.161-2.b14.el7.x86_64
            [root@zte jdk1.8.0_131]# rpm -e java-1.8.0-openjdk-1.8.0.161-2.b14.el7.x86_64 tzdata-java-2018c-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch javapackages-tools-3.4.1-11.el7.noarch java-1.8.0-openjdk-headless-1.8.0.161-2.b14.el7.x86_64 --nodeps


            vim /etc/profile
                # ---------------------------- jwb profile ------------------
                
                # jdk
                export JAVA_HOME=/opt/modules/jdk1.8.0_131
                export PATH=:${JAVA_HOME}/bin:$PATH

        （3）将jdk的包传给另外两台机器
             scp -r jdk1.8.0_131/ zte3:$PWD

    2）安装MySQL
        ambari安装会将安装等信息写入数据库，建议使用自己安装的Mysql数据库，也可以不安装而使用默认数据库PostgreSQL。如果安装Hive、Oozie，也可使用MySQL数据库

        #注意：在线安装时，centos7.x已经将MySQL换为MariaDB

        1>> 先卸载MySQL和MariaDB的包
            # 要卸载干净，否则后面安装rpm会出现冲突
            rpm -qa | grep mysql 
            rpm -qa | grep mariadb

        2>> 先下载对应版本的rpm-bundle.tar，然后解压            
            tar -xvf MySQL-5.6.46-1.el7.x86_64.rpm-bundle.tar 

            ------------------------------------------------------------------------------
              40310724 9月  29 21:42 MySQL-client-5.6.46-1.el7.x86_64.rpm
               6653248 9月  29 21:42 MySQL-devel-5.6.46-1.el7.x86_64.rpm
             101802416 9月  29 21:42 MySQL-embedded-5.6.46-1.el7.x86_64.rpm
              90592828 9月  29 21:42 MySQL-server-5.6.46-1.el7.x86_64.rpm
               4247088 9月  29 21:42 MySQL-shared-5.6.46-1.el7.x86_64.rpm
               2299664 9月  29 21:42 MySQL-shared-compat-5.6.46-1.el7.x86_64.rpm
              66404888 9月  29 21:42 MySQL-test-5.6.46-1.el7.x86_64.rpm
            ------------------------------------------------------------------------------


        3>> 创建MySQL用户和组
            groupadd -g 1000 mysql
            useradd mysql -g mysql -p mysql
        
        4>> 安装
            rpm -ivh MySQL-server-5.6.46-1.el7.x86_64.rpm
            rpm -ivh MySQL-client-5.6.46-1.el7.x86_64.rpm
            rpm -ivh MySQL-devel-5.6.46-1.el7.x86_64.rpm
            rpm -ivh MySQL-shared-5.6.46-1.el7.x86_64.rpm

        5>> 启动
                service mysql start
            使用随机密码登录
                find / -name .mysql_secret
                mysql -uroot -p

        6>> 修改密码
            登陆成功后，修改密码，否则无法选择数据库进行操作
                SET PASSWORD=password('123456');
                FLUSH PRIVILEGES;
            至此，便可以进行mysql操作了；

    3）创建MySQL用户和库备用
            # 新建ambari数据库
            create database ambari default charset utf8 collate utf8_general_ci; 
            #新建ambari用户
            CREATE USER 'ambari'@'%' IDENTIFIED BY '123456';
            #分配ambari权限
            GRANT ALL PRIVILEGES ON ambari.* TO 'ambari'@'%';
            FLUSH PRIVILEGES;

            # 新建hive数据库
            create database hive default charset utf8 collate utf8_general_ci; 
            #新建hive用户
            CREATE USER 'hive'@'%' IDENTIFIED BY '123456';
            #分配hive权限
            GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'%';
            FLUSH PRIVILEGES;

            # 新建oozie数据库
            create database oozie default charset utf8 collate utf8_general_ci; 
            #新建oozie用户
            CREATE USER 'oozie'@'%' IDENTIFIED BY '123456';
            #分配oozie权限
            GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
            FLUSH PRIVILEGES;

            # 新建hue数据库
            create database hue default charset utf8 collate utf8_general_ci; 
            #新建hue用户
            CREATE USER 'hue'@'%' IDENTIFIED BY '123456';
            #分配hue权限
            GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%';
            FLUSH PRIVILEGES;

5.安装Ambari
    1）新建文件夹：
            mkdir /var/www/html/ambari
            mkdir /var/www/html/hdp
            mkdir /var/www/html/hdp-utils
            mkdir /var/www/html/hdp-gpl


    2）下载安装包并解压（这里都是在外面下载好了再上传到/opt/softs中，如果想直接在机器里下载则用wget）
        1>> HDP-3.0.1.0-centos7-rpm.tar.gz
            wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.1.0/HDP-3.0.1.0-centos7-rpm.tar.gz
            tar -xvf HDP-3.0.1.0-centos7-rpm.tar.gz -C /opt/modules

        2>> HDP-UTILS-1.1.0.22-centos7.tar.gz
                http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
                tar -xvf HDP-UTILS-1.1.0.22-centos7.tar.gz -C /opt/modules

        3>> ambari-2.7.1.0-centos7.tar.gz
                wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.1.0/ambari-2.7.1.0-centos7.tar.gz
                tar -xvf ambari-2.7.1.0-centos7.tar.gz -C /opt/modules

        4>> HDP-GPL-3.0.1.0-centos7-gpl.tar.gz
                wget http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.0.1.0/HDP-GPL-3.0.1.0-centos7-gpl.tar.gz
                tar -xvf HDP-GPL-3.0.1.0-centos7-gpl.tar.gz -C /opt/modules


        # 挂载新磁盘补充
            由于最初的时候分配的是20G硬盘，所以很快就达到了，关闭虚拟机，新添加一块磁盘
            查看磁盘情况
                df -lh  
                fdisk -l  查看linux上的设备（cpu、硬盘都是一个个设备） 
            开始分区 
                fdisk /dev/sdb   这个命令执行后依次输 n  p  1  回车  回车  w 
                fdisk -l 
                mkfs -t ext4 -c /dev/sdb1   格式化我们的分区 
            将我们的分区挂载到/data目录下 
                mkdir /data
                mount -t ext4 /dev/sdb1 /data 
                df -lh 
            将我们的挂载磁盘设置开机启动，避免开机之后挂载的磁盘就没了 
                echo "/dev/sdb1 /data ext4 defaults 0 0" >> /etc/fstab    
            


    3）下载Ambari.repo并修改

        移动文件：
            mv ambari/centos7/2.7.1.0-169/* /var/www/html/ambari/
            mv HDP/centos7/3.0.1.0-187/* /var/www/html/hdp/
            mv HDP-UTILS/centos7/1.1.0.22/* /var/www/html/hdp-utils/
            mv HDP-GPL/centos7/3.0.1.0-187/* /var/www/html/hdp-gpl/
        
        
        下载：ambari.repo 并修改
            wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.1.0/ambari.repo  （任意目录，可以/opt/files）
            然后修改
                baseurl=http://zte1/ambari
                gpgkey=http://zte1/ambari/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

        下载hdp.repo 并修改
            wget http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.0.1.0/hdp.repo
            然后修改：
                [HDP-3.0.1.0]
                baseurl=http://zte1/hdp
                gpgkey=http://zte1/hdp/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
                
                [HDP-UTILS-1.1.0.22]
                baseurl=http://zte1/hdp-utils
                gpgkey=http://zte1/hdp-utils/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins



        复制repo，将修改后的ambari.repo和hdp.repo复制到/etc/yum.repos.d
            cp ambari.repo /etc/yum.repos.d
            cp hdp.repo /etc/yum.repos.d

        之后再发送到zte2和zte3
            scp ambari.repo hdp.repo zte2:/etc/yum.repos.d/
            scp ambari.repo hdp.repo zte2:/etc/yum.repos.d/

            （若是两个节点在安装agent时出现：没有可用软件包 ambari-agent。则是这个没配好）

        重启httpd
            systemctl restart httpd

        更新yum缓存
            yum clean all
            yum makecache
        
        查看ambari源
            yum list | grep ambari

        安装MySQL驱动
            wget https://cdn.mysql.com/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
            tar -xvf mysql-connector-java-5.1.46.tar.gz -c /data/modules
        
        设置ambari的MySQL驱动：（需启动ambari-server后执行）移动驱动到/usr/share/java下后面备用
            cd mysql-connector-java-5.1.46
            cp mysql-connector-java-5.1.46.jar /usr/share/java/

    4）安装ambari
        # 安装ambari-server
            在选择的主节点上安装ambari-server，每个节点（包括主节点）上安装ambari-agent
            yum install ambari-server -y

        >> 设置Ambari server
            ambari-server setup

            设置JAVA_HOME路径（在1.9中配置的路径）：
            Customize user account for ambari-server daemon [y/n] (n)? @@@y
            Enter user account for ambari-server daemon (root): @@@root
            [@@@ 2] Custom JDK
            Path to JAVA_HOME: @@@ /opt/modules/jdk1.8.0_131
            Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? @@@ y
            Enter advanced database configuration [y/n] (n)? @@@ y
            [@@@ 3] - MySQL / MariaDB
            Hostname (localhost): @@@ zte1
            Port (3306): @@@ 3306
            Database name (ambari): @@@ ambari
            Username (ambari): @@@ ambari
            Enter Database Password (bigdata): @@@ 123456
            Enter full path to custom jdbc driver: @@@ /usr/share/java/mysql-connector-java-5.1.46.jar
            Proceed with configuring remote database connection properties [y/n] (y)? @@@ y


            
            设置ambari内置数据库


        >> mysql 添加ambari表头
            登陆mysql mysql -u ambari -p123456
        
            use ambari;
            source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

        # 启动ambari
            ambari-server start
            ambari-server stop #停止ambari-server



        >> 对每个节点安装Ambari-agent
            每个节点都需要安装：
                yum install ambari-agent -y
            
            设置Ambari Server Host Name：
            编辑ambari-agent配置文件，全部设置为安装过Ambari-server 的主机名：
                vim /etc/ambari-agent/conf/ambari-agent.ini
            分别将hostname修改为zte1

        可以修改完zte1之后发送到zte2、zte3：
            scp /etc/ambari-agent/conf/ambari-agent.ini root@zte2:/etc/ambari-agent/conf/
            scp /etc/ambari-agent/conf/ambari-agent.ini root@zte3:/etc/ambari-agent/conf/

        # 启动ambari agent
            ambari-agent start

            相关的命令有以下：
                ambari-agent status # 检查agent服务运行状态
                ambari-agent stop # 停止agent服务


6.安装HDP

    1）登录到Apache Ambari
        在启动Ambari 服务之后，使用web浏览器，打开Ambari Web。在浏览器的地址栏输入：
            http://192.168.159.11:8080
            默认用户名密码为：admin admin

    2）给集群命名
        点击Launch Install Wizard，输入集群名称
        

    3）选择HDP版本
        移除除了你需要的，只留下rehat7
        选择本地源和系统版本，填写对应地址：
            HDP-3.1：                http://zte1/hdp
            HDP-3.1-GPL：            http://zte1/hdp-gpl
            HDP-UTILS-1.1.0.22：        http://zte1/hdp-utils


        ------------------------------------------
            Accumulo         1.7.0
            Infra Solr         0.1.0
            Ambari Metrics     0.1.0
            Atlas             1.0.0
            Druid             0.12.1
            HBase             2.0.0
            HDFS             3.1.1
            Hive             3.1.0
            Kafka             1.1.1
            Knox             1.0.0
            Log Search         0.5.0
            Oozie             4.3.1
            Pig             0.16.0
            Ranger             1.1.0
            Ranger KMS         1.1.0
            SmartSense         1.5.0.2.7.1.0-169
            Spark2             2.3.1
            Sqoop             1.4.7
            Storm             1.2.1
            Superset         0.23.0
            Tez             0.9.1
            YARN             3.1.1
            Zeppelin Notebook     0.8.0
            ZooKeeper         3.4.6
        ---------------------------------------------

    4）安装选项
        在“Target Hosts”里填上主机名，主机名一定要是全限定名。一行一个主机名。
            zte.hdp.bigdata1
            zte.hdp.bigdata2
            zte.hdp.bigdata3

        在“Host Registration information”，选择Perform manual registration on hosts and do not use SSH，使用这种方式需要注意：需要提前在host上安装好ambari-agent，并启动服务。
        在主机 cat /root/.ssh/id_rsa 复制粘贴过去（注意全部都粘贴）

        Host Checks时发现一个警告：Service Issues (1)
        The following services should be up Service ntpd or chronyd Not running on 1 hosts
        解决方案:NTP服务要在三个节点都    启动
            zte1/zte2/zte3：        service ntpd start


        ## 确保Host Checks中没有一个报错或者警告，必须解决，防止后期集群出问题


    5）确认主机和选择服务
        根据需求自主选择需要安装的组件，示例安装以下内容：
        HDFS、YARN + MapReduce2、Hive、HBase、ZooKeeper、Ambari Metrics、Kafka、Spark2、Slider


        -----------------------------------------------------------------------------------------------------------------------------------------------------
        Choose File System    Choose which file system you want to install on your cluster.
            Service             Version         Description
            HDFS                 3.1.1         Apache Hadoop Distributed File System

        Choose Services    Choose which services you want to install on your cluster.
            Service             Version         Description
            YARN + MapReduce2     3.1.1         Apache Hadoop NextGen MapReduce (YARN)
            Tez                 0.9.1         Tez is the next generation Hadoop Query Processing framework written on top of YARN.
            Hive                 3.1.0         Data warehouse system for ad-hoc queries & analysis of large datasets and table & storage management service
            HBase                 2.0.0         Non-relational distributed database and centralized service for configuration management & synchronization
            Pig                 0.16.0         Scripting platform for analyzing large datasets
            Sqoop                 1.4.7         Tool for transferring bulk data between Apache Hadoop and structured data stores such as relational databases
            Oozie                 4.3.1         System for workflow coordination and execution of Apache Hadoop jobs. This also includes the installation of the optional Oozie Web Console which relies on and will install the ExtJS Library.
            ZooKeeper             3.4.6         Centralized service which provides highly reliable distributed coordination
            Storm                 1.2.1         Apache Hadoop Stream processing framework
            Accumulo             1.7.0         Robust, scalable, high performance distributed key/value store.
            Infra Solr             0.1.0         Core shared service used by Ambari managed components.
            Ambari Metrics         0.1.0         A system for metrics collection that provides storage and retrieval capability for metrics collected from the cluster
            Atlas                 1.0.0         Atlas Metadata and Governance platform
            Kafka                 1.1.1         A high-throughput distributed messaging system
            Knox                 1.0.0         Provides a single point of authentication and access for Apache Hadoop services in a cluster
            Log Search             0.5.0         Log aggregation, analysis, and visualization for Ambari managed services. This service is Technical Preview.
            Ranger                 1.1.0         Comprehensive security for Hadoop
            Ranger KMS             1.1.0         Key Management Server
            SmartSense             1.5.0.2.7.1.0-169     SmartSense - Hortonworks SmartSense Tool (HST) helps quickly gather configuration, metrics, logs from common HDP services that aids to quickly troubleshoot support cases and receive cluster-specific recommendations.
            Spark2                 2.3.1         Apache Spark 2.3 is a fast and general engine for large-scale data processing.
            Zeppelin Notebook     0.8.0         A web-based notebook that enables interactive data analytics. It enables you to make beautiful data-driven, interactive and collaborative documents with SQL, Scala and more.
            Druid                 0.12.1         A fast column-oriented distributed data store.
            Superset             0.23.0         Superset is a data exploration platform designed to be visual, intuitive and interactive. This service is Technical Preview.
        ------------------------------------------------------------------------------------------------------------------------------------------------------


    6）配置每个节点的服务

    7）用户名密码
        Grafana Admin    admin/123456
        Hive Database     hive/123456
        Log Search    ambari_logsearch_admin/123456
        Oozie Database    oozie/123456
        Activity Explorer's Admin      N/A / 123456


    8）数据库的配置
        hive：The MySQL JDBC driver has not been set. Please ensure that you have executed 'ambari-server setup --jdbc-db=mysql --jdbc-driver=/path/to/jdbc_driver'.

            ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-5.1.46.jar

    9）启动时，端口占用导致启动失败
        lsof -i:53
        kill -9 [pid]