说明:此文档是讲解ambari离线安装HDP的方法,这里的离线安装说的是ambari组件、HDP组件这些大的文件从网站上下载好避免网络不好中间中断,但是有些小文件如PostgreSQL、系统初始化yum更新、mysql之类的包也需要网络,当然可以本地yum源,只是麻烦些,建议还是需要服务器可以访问外网。
在线安装可参考:http://blog.csdn.net/levy_cui/article/details/59108546 因为线上环境是3年前安装的,所以版本比较低,新安装可以尝试较新的版本,安装步骤是一样的。
AMBARI-2.4.0.1-centos6.tar.gz
包含HDP 2.2 - HDP 2.5版本
ambari-2.1.0-centos6.tar.gz
一、下载:
cd /usr/local/src/
http://s3.amazonaws.com/public-repo-1.hortonworks.com/ambari/centos6/ambari-2.1.0-centos6.tar.gz (482M)
http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/2.1-latest/HDP-2.1-latest-centos6-rpm.tar.gz (2.6G)
http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz (694M)
当这些文件下载后,将这些压缩包的文件解压到 /var/www/html中,为了方便管理建议在该目录下创建一个external目录
使用tar命令解压缩:
yum install httpd
mkdir -p /var/www/html/external //新建目录,将下载的文件解压到/var/www/html/external中
tar -xvf /usr/local/src/HDP-2.1-latest-centos6-rpm.tar.gz -C /var/www/html/external/
tar -xvf /usr/local/src/HDP-UTILS-1.1.0.20-centos6.tar.gz -C /var/www/html/external/
tar -xvf /usr/local/src/ambari-2.1.0-centos6.tar.gz -C /var/www/html/external/
之后,在/etc/yum.repos.d创建三个repo文件
#vi /etc/yum.repos.d/ambari.repo
[Updates-ambari-2.1.0]
name=ambari-2.1.0 - Updates
baseurl=http://hadooptest76.bj/external/ambari-2.1.0/centos6
gpgcheck=1
gpgkey=http://hadooptest76.bj/external/ambari-2.1.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
#vi /etc/yum.repos.d/hdp.repo
[HDP-2.1.10.0]
name=HDP Version - HDP-2.1.10.0
baseurl=http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0
gpgcheck=1
gpgkey=http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://hadooptest76.bj/external/HDP-UTILS-1.1.0.20/repos/centos6
gpgcheck=1
gpgkey=http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
#vi /etc/yum.repos.d/hdp-util.repo
[HDP-UTILS-1.1.0.20]
name=Hortonworks Data Platform Version - HDP-UTILS-1.1.0.20
baseurl= http://hadooptest76.bj/external/HDP-UTILS-1.1.0.20/repos/centos6
gpgcheck=1
enabled=1
priority=1
之后,将写好的文件,发送至其他节点上(这里只举一个例子,需要scp到所有节点上)
scp ambari.repo root@Test02:/etc/yum.repo.d/
scp hdp.repo root@Test02:/etc/yum.repo.d/
scp hdp-util.repo root@Test02:/etc/yum.repo.d/
二、在ambari服务节点运行
yum install ambari-server
下载或者从其他节点scp jdk(我这里之前已经下载过了所有直接scp一份到ambari节点上)
scp /var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz 10.11.32.76:/var/lib/ambari-server/resources/
三、安装MySQL服务
1)安装
通过yum安装mysql:
[root@cluster-02 ~]$ yum install mysql-server
启动mysql服务
[root@cluster-02 ~]$ service mysqld start
安装MySQL JDBC Connector:
[root@cluster-02 root]$ yum install mysql-connector-java
2)为root用户设置新密码等初始化工作
[root@cluster-01 ~]$ /usr/bin/mysql_secure_installation
3)为ambari创建数据库,配置相应用户和权限
[root@cluster-02 ~]# mysql -u root -p
mysql> create database ambari;
Query OK, 1 row affected (0.00 sec)
mysql> use ambari;
Database changed
4)hive库
mysql> create database hive;
mysql> create user "hive" identified by "hive123";
mysql> grant all privileges on hive.* to hive;
mysql> flush privileges;
5)启动
mysql设置为开机自启动
[root@cluster-05 root]$ chkconfig mysqld on
[root@cluster-05 root]$ chkconfig --list
mysqld 0:off 1:off 2:on 3:on 4:on 5:on 6:off
6)导入数据库
注意,此刻需要切换到mysql中执行相应脚本操作。
需要登录到mysql数据库,执行以下脚本Ambari-DDL-MySQL-CREATE.sql
[root@ambari ~]# mysql -u root -p
mysql> use ambari;
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
执行脚本,成功。
ambari权限设置
CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
FLUSH PRIVILEGES;
四、ambari server安装
[root@Ambari yum.repos.d]# ambari-server setup
Using python /usr/bin/python2.6
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? (直接回车)
Adjusting ambari-server permissions and ownership...
Checking firewall...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)? (输入y直接回车)
Checking JDK...
[1] - Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[2] - Oracle JDK 1.6 + Java Cryptography Extension (JCE) Policy Files 6
[3] - Custom JDK
==============================================================================
Enter choice (1):1 (直选择1 根据版本选择)
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at
http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files
manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? (直接回车)
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz
JDK distribution size is 142376665 bytes
jdk-7u67-linux-x64.tar.gz... 100% (135.8 MB of 135.8 MB)
Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz
Installing JDK to /usr/jdk64
Successfully installed JDK to /usr/jdk64/jdk1.7.0_67
Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip to /var/lib/ambari-
server/resources/UnlimitedJCEPolicyJDK7.zip
Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/UnlimitedJCEPolicyJDK7.zip
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y (y回车)
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
==============================================================================
Enter choice (1): 3
Hostname (localhost): (直接回车)
Port (3306): (直接回车)
Database Name (ambari): (直接回车)
Username (ambari): (直接回车)
Enter Database Password (bigdata): (输入ambari密码,我这里是默认bigdata)
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-
CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? (直接回车)
Copying JDBC drivers to server resources...
Extracting system views...
ambari-admin-1.7.0.169.jar
..
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
[root@ambari ~]# ambari-server start
Using python /usr/bin/python2.6
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................
Ambari Server 'start' completed successfully.
使用浏览器打开IP:8080, 输入admin/admin 即可访问ambari管理界面开始HDP集群安装。
五、安装HDP(包含hadoop、zookeeper、hive等等)
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6
修改为:
http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0/
http://hadooptest76.bj/external/HDP-UTILS-1.1.0.20/repos/centos6
配置hive的时候需要安装mysql-connector-java.jar
find / -name mysql-connector-java.jar
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
其余内容可参考http://www.cnblogs.com/tgzhu/tag/HDP/ 写的挺详细了
参考:
http://www.cnblogs.com/roger888/p/5505731.html
http://blog.csdn.net/hyd38888/article/details/48136461
http://blog.csdn.net/poisonchry/article/details/37726663
在线安装可参考:http://blog.csdn.net/levy_cui/article/details/59108546 因为线上环境是3年前安装的,所以版本比较低,新安装可以尝试较新的版本,安装步骤是一样的。
AMBARI-2.4.0.1-centos6.tar.gz
包含HDP 2.2 - HDP 2.5版本
ambari-2.1.0-centos6.tar.gz
包含HDP 2.1 - HDP 2.4版本
准备工作:
安装集群前需要进行系统初始化设置,如主机hosts、ssh免秘钥、文件句柄打开数、防火墙selinux关闭等等,网上很多自行搜索。
一、下载:
cd /usr/local/src/
http://s3.amazonaws.com/public-repo-1.hortonworks.com/ambari/centos6/ambari-2.1.0-centos6.tar.gz (482M)
http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/2.1-latest/HDP-2.1-latest-centos6-rpm.tar.gz (2.6G)
http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz (694M)
当这些文件下载后,将这些压缩包的文件解压到 /var/www/html中,为了方便管理建议在该目录下创建一个external目录
使用tar命令解压缩:
yum install httpd
mkdir -p /var/www/html/external //新建目录,将下载的文件解压到/var/www/html/external中
tar -xvf /usr/local/src/HDP-2.1-latest-centos6-rpm.tar.gz -C /var/www/html/external/
tar -xvf /usr/local/src/HDP-UTILS-1.1.0.20-centos6.tar.gz -C /var/www/html/external/
tar -xvf /usr/local/src/ambari-2.1.0-centos6.tar.gz -C /var/www/html/external/
之后,在/etc/yum.repos.d创建三个repo文件
#vi /etc/yum.repos.d/ambari.repo
[Updates-ambari-2.1.0]
name=ambari-2.1.0 - Updates
baseurl=http://hadooptest76.bj/external/ambari-2.1.0/centos6
gpgcheck=1
gpgkey=http://hadooptest76.bj/external/ambari-2.1.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
#vi /etc/yum.repos.d/hdp.repo
[HDP-2.1.10.0]
name=HDP Version - HDP-2.1.10.0
baseurl=http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0
gpgcheck=1
gpgkey=http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://hadooptest76.bj/external/HDP-UTILS-1.1.0.20/repos/centos6
gpgcheck=1
gpgkey=http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
#vi /etc/yum.repos.d/hdp-util.repo
[HDP-UTILS-1.1.0.20]
name=Hortonworks Data Platform Version - HDP-UTILS-1.1.0.20
baseurl= http://hadooptest76.bj/external/HDP-UTILS-1.1.0.20/repos/centos6
gpgcheck=1
enabled=1
priority=1
之后,将写好的文件,发送至其他节点上(这里只举一个例子,需要scp到所有节点上)
scp ambari.repo root@Test02:/etc/yum.repo.d/
scp hdp.repo root@Test02:/etc/yum.repo.d/
scp hdp-util.repo root@Test02:/etc/yum.repo.d/
二、在ambari服务节点运行
yum install ambari-server
下载或者从其他节点scp jdk(我这里之前已经下载过了所有直接scp一份到ambari节点上)
scp /var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz 10.11.32.76:/var/lib/ambari-server/resources/
三、安装MySQL服务
1)安装
通过yum安装mysql:
[root@cluster-02 ~]$ yum install mysql-server
启动mysql服务
[root@cluster-02 ~]$ service mysqld start
安装MySQL JDBC Connector:
[root@cluster-02 root]$ yum install mysql-connector-java
2)为root用户设置新密码等初始化工作
[root@cluster-01 ~]$ /usr/bin/mysql_secure_installation
3)为ambari创建数据库,配置相应用户和权限
[root@cluster-02 ~]# mysql -u root -p
mysql> create database ambari;
Query OK, 1 row affected (0.00 sec)
mysql> use ambari;
Database changed
4)hive库
mysql> create database hive;
mysql> create user "hive" identified by "hive123";
mysql> grant all privileges on hive.* to hive;
mysql> flush privileges;
5)启动
mysql设置为开机自启动
[root@cluster-05 root]$ chkconfig mysqld on
[root@cluster-05 root]$ chkconfig --list
mysqld 0:off 1:off 2:on 3:on 4:on 5:on 6:off
6)导入数据库
注意,此刻需要切换到mysql中执行相应脚本操作。
需要登录到mysql数据库,执行以下脚本Ambari-DDL-MySQL-CREATE.sql
[root@ambari ~]# mysql -u root -p
mysql> use ambari;
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
执行脚本,成功。
ambari权限设置
CREATE USER 'ambari'@'%' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%';
FLUSH PRIVILEGES;
四、ambari server安装
[root@Ambari yum.repos.d]# ambari-server setup
Using python /usr/bin/python2.6
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? (直接回车)
Adjusting ambari-server permissions and ownership...
Checking firewall...
WARNING: iptables is running. Confirm the necessary Ambari ports are accessible. Refer to the Ambari documentation for more details on ports.
OK to continue [y/n] (y)? (输入y直接回车)
Checking JDK...
[1] - Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[2] - Oracle JDK 1.6 + Java Cryptography Extension (JCE) Policy Files 6
[3] - Custom JDK
==============================================================================
Enter choice (1):1 (直选择1 根据版本选择)
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at
http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files
manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? (直接回车)
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz
JDK distribution size is 142376665 bytes
jdk-7u67-linux-x64.tar.gz... 100% (135.8 MB of 135.8 MB)
Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-7u67-linux-x64.tar.gz
Installing JDK to /usr/jdk64
Successfully installed JDK to /usr/jdk64/jdk1.7.0_67
Downloading JCE Policy archive from http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip to /var/lib/ambari-
server/resources/UnlimitedJCEPolicyJDK7.zip
Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/UnlimitedJCEPolicyJDK7.zip
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y (y回车)
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
==============================================================================
Enter choice (1): 3
Hostname (localhost): (直接回车)
Port (3306): (直接回车)
Database Name (ambari): (直接回车)
Username (ambari): (直接回车)
Enter Database Password (bigdata): (输入ambari密码,我这里是默认bigdata)
Copying JDBC drivers to server resources...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-
CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? (直接回车)
Copying JDBC drivers to server resources...
Extracting system views...
ambari-admin-1.7.0.169.jar
..
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
[root@ambari ~]# ambari-server start
Using python /usr/bin/python2.6
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start....................
Ambari Server 'start' completed successfully.
使用浏览器打开IP:8080, 输入admin/admin 即可访问ambari管理界面开始HDP集群安装。
五、安装HDP(包含hadoop、zookeeper、hive等等)
离线需要在界面中源路径配置相关内容:
Use Local Repository
http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.2.6.0http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6
修改为:
http://hadooptest76.bj/external/HDP/centos6/2.x/updates/2.1.10.0/
http://hadooptest76.bj/external/HDP-UTILS-1.1.0.20/repos/centos6
配置hive的时候需要安装mysql-connector-java.jar
find / -name mysql-connector-java.jar
ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
其余内容可参考http://www.cnblogs.com/tgzhu/tag/HDP/ 写的挺详细了
参考:
http://www.cnblogs.com/roger888/p/5505731.html
http://blog.csdn.net/hyd38888/article/details/48136461
http://blog.csdn.net/poisonchry/article/details/37726663