Centos7下Cloudera Manager 5离线部署 CDH5
文章目录
1.先决条件
2.准备工作
-
3台16核32G内存260G硬盘CentOS 7机器
机器规划：
HostName 角色 IP cdh001 CM节点，NameNode，Active Monitor,DataNode 192.168.31.151 cdh002 DataNode、 JobTracker 192.168.31.152 cdh003 DataNode、 JobTracker 192.168.31.153 -
CM离线包
-
CDH离线包
- CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel
- CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1 (注意下载后需要重命名为**.sha,否则安装parcel的时候会在线下载)
- manifest.json
-
JDK1.8u162
-
JDBC
-
操作系统依赖
-
psmisc
CM客户端和服务端启动脚本会使用pstree命令，不安装会报找不到命令
-
libxslt-devel
不安装的话Hue安装时测试MySQL会报错
# 所有节点安装 yum install -y psmisc libxslt-devel wget
-
3.CM安装
3.1.网络配置
-
修改主机名
# 分别在对应机器上执行 hostnamectl set-hostname cdh01 hostnamectl set-hostname cdh02 hostnamectl set-hostname cdh03 -
配置hosts文件 (所有节点)
vi /etc/hosts 192.168.31.151 cdh01 192.168.31.152 cdh02 192.168.31.153 cdh03 -
配置ssh免密登录
# 在cdh01上执行 ssh-keygen # 一路回车默认在~/.ssh目录下生成RSA密钥对id_rsa(私钥)，id_rsa.pub（公钥） # 添加公钥认证 ssh-copy-id -i cdh01 ssh-copy-id -i cdh02 ssh-copy-id -i cdh03 # 输入机器用户密码 # 拷贝密钥对到各个机器上去 scp -r ~/.ssh/ cdh01:~/ scp -r ~/.ssh/ cdh02:~/ scp -r ~/.ssh/ cdh03:~/ -
关闭防火墙 （所有节点）
# 检查防火墙状态 systemctl status firewalld # 关闭防火墙 systemctl stop firewalld # 设置防火墙开机不自启 systemctl disable firewalld -
关闭selinux（所有节点）
# 临时关闭selinux setenforce 0 # 设置开机不自启 vi /etc/selinux/config # 修改"SELINUX=enforcing"为"SELINUX=disabled" SELINUX=disabled # 查看 SELINUX 状态： /usr/sbin/sestatus –v # SELinux status: enabled（enabled：开启；disabled：关闭）
3.2.JDK安装
-
官方JDK安装要求
- The JDK must be 64-bit. Do not use a 32-bit JDK.
- The installed JDK must be a supported version as documented in CDH and Cloudera Manager Supported JDK Versions.
- Oracle JDK 7 is supported across all versions of Cloudera Manager 5 and CDH 5.
- Oracle JDK 8 is supported in C5.3.x and higher.
- Oracle JDK 9 is not supported in any Cloudera Manager or CDH version.
- The same version of the Oracle JDK must be installed on each cluster host.
- The JDK must be installed at /usr/java/jdk-version.
Important:The RHEL-compatible and Ubuntu operating systems supported by Cloudera Enterprise all use AES-256 encryption by default for tickets. To support AES-256 bit encryption in JDK versions lower than 1.8u161, you must install the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy File on all cluster and Hadoop user machines. Cloudera Manager can automatically install the policy files, or you can install them manually. For JCE Policy File installation instructions, see the README.txtfile included in the jce_policy-x.zip file. JDK 1.8u161 and higher enable unlimited strength encryption by default, and do not require policy files.On SLES platforms, do not install or try to use the IBM Java version bundled with the SLES distribution. CDH does not run correctly with that version.
根据官方指南因此本次安装选择JDK充分测试过且支持的最高版本JDK1.8u162
# 上传下载的jdk到cdh01机器上 # 解压缩 # 移动到/opt/java目录下 mkdir /usr/java/ tar zxvf jdk-8u162-linux-x64.tar.gz -C /usr/java/ # 配置环境变量 vi /etc/profile # 在最末尾加上 # for jdk export JAVA_HOME=/usr/java/jdk1.8.0_162 export JRE_HOME=${JAVA_HOME}/jre export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib export PATH=${JAVA_HOME}/bin:$PATH # 生效环境变量 source /etc/profile # 检查是否生效 java -version # 将jdk拷贝到其他节点上去 scp -r /usr/java/ cdh02:/usr/ scp -r /usr/java/ cdh03:/usr/ # 拷贝环境变量到其他节点上去 scp /etc/profile cdh02:/etc/ scp /etc/profile cdh03:/etc/ # 在其他节点检查jdk环境是否生效 source /etc/profile java -version
3.3.配置ntp
-
安装ntp（所有节点）
yum install -y ntp -
配置
-
cdh01节点
# cdh01主节点配置 vi /etc/ntp.conf # 注释默认的server,添加以下server server 210.72.145.44 perfer # 中国国家受时中心 server 202.112.10.36 # 1.cn.pool.ntp.org server 59.124.196.83 # 0.asia.pool.ntp.org server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10 -
cdh02、cdh03节点
vi /etc/ntp.conf ## 子节点把所有server行注释掉，添加下面一行 server cdh01 -
所有节点
systemctl start ntpd systemctl enable ntpd ntpstatsynchronised to NTP server (192.168.31.151) at stratum 12
time correct to within 909 ms
polling server every 64 s
-
3.4.cdh01节点安装配置MySQL
-
安装
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm rpm -ivh mysql-community-release-el7-5.noarch.rpm yum update yum install mysql-server systemctl start mysqld -
配置
systemctl stop mysqld cp /etc/my.cnf /etc/my.cnf.default vi /etc/my.cnf # 删除原有配置，并修改为如下建议配置[mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: symbolic-links = 0 key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 550 #expire_logs_days = 10 #max_binlog_size = 100M #log_bin should be on a disk with enough free space. #Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your #system and chown the specified folder to the mysql user. log_bin=/var/lib/mysql/mysql_binary_log #In later versions of MySQL, if you enable the binary log and do not set #a server_id, MySQL will not start. The server_id must be unique within #the replicating group. server_id=1 binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M # InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid sql_mode=STRICT_ALL_TABLES# 设置开机自启 systemctl enable mysqld # 启动 systemctl start mysqld -
设置root密码
/usr/bin/mysql_secure_installation [...] Enter current password for root (enter for none): OK, successfully used password, moving on... [...] Set root password? [Y/n] Y New password: Re-enter new password: Remove anonymous users? [Y/n] Y [...] Disallow root login remotely? [Y/n] N [...] Remove test database and access to it [Y/n] Y [...] Reload privilege tables now? [Y/n] Y All done! -
安装配置MySQL JDBC 驱动
下载 .tar.gz格式的文件地址：http://www.mysql.com/downloads/connector/j/5.1.html
wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz tar zxvf mysql-connector-java-5.1.47.tar.gz mkdir -p /usr/share/java/ cp mysql-connector-java-5.1.47/mysql-connector-java-5.1.47-bin.jar \ /usr/share/java/mysql-connector-java.jar scp -r /usr/share/java/ cdh02:/usr/share/ scp -r /usr/share/java/ cdh03:/usr/share/ -
创建对应服务数据库
mysql -uroot -p mysql>CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue'; CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive'; CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry'; CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie'; CREATE DATABASE am DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON am.* TO 'am'@'%' IDENTIFIED BY 'am';
3.5. 创建用户 cloudera-scm （ 所有 节点）
useradd --system --home=/opt/cm-5.15.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
3.6.解压CM包
# cdh01节点
tar -zxvf cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz -C /opt/
mv CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel manifest.json /opt/cloudera/parcel-repo/
chown -R cloudera-scm:cloudera-scm /opt/cm-5.15.1/
mkdir -p /opt/cloudera/parcel-repo
chown -R cloudera-scm:cloudera-scm /opt/cloudera/
mkdir /var/lib/cloudera-scm-server
chown -R cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server
3.7.修改agent配置
# cdh01节点
vi /opt/cm-5.15.1/etc/cloudera-scm-agent/config.ini
server_host=cdh01
3.8.拷贝CM包到所有节点
# cdh01节点
scp -r /opt/cm-5.15.1/ cdh02:/opt/
scp -r /opt/cm-5.15.1/ cdh03:/opt/
# cdh02、cdh03节点
mkdir -p /opt/cloudera/parcel-repo
chown -R cloudera-scm:cloudera-scm /opt/cloudera/
3.9.cdh01节点初始化数据库
# cdh01节点
/opt/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -p123456 --scm-host localhost scm scm scm
3.10.启动服务
# cdh01节点启动cm服务端
/opt/cm-5.15.1/etc/init.d/cloudera-scm-server start
# 查看启动日志
tail -f /opt/cm-5.15.1/log/cloudera-scm-server/cloudera-scm-server.log
# 查看到如下日志服务启动完成
com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
# cdh01、cdh02、cdh03节点启动客户端
/opt/cm-5.15.1/etc/init.d/cloudera-scm-agent start
3.11.初始化CM
-
访问CM web界面
用户：admin
密码：admin
-
配置CM
此处会报两个错误：内存交换空间设置和启用了透明大页面压缩，点击【查看详细信息】查看需要修改设置的主机
-
将机器设置成cloudera官方建议的配置
sysctl vm.swappiness=10 vi /etc/sysctl.conf # /etc/sysctl.conf加入或修改vm.swappiness配置 vm.swappiness=10 # 使配置生效 sysctl -pecho never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.local echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.local cat /etc/rc.local #!/bin/bash # THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES # # It is highly advisable to create own systemd services or udev rules # to run scripts during boot instead of using this file. # # In contrast to previous versions due to parallel execution during boot # this script will NOT be run after all other services. # # Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure # that this script will be executed during boot. touch /var/lock/subsys/local echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled修改完成重新运行检查主机检查，此时警告消除点击【完成】。
4.安装CDH服务
- 选择服务
- 配置角色
- 设置数据库连接
- 集群设置
- 开始安装
- 完成安装
3万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
