CDH5.16.2真正的离线部署
参考视频:若泽大数据
-
MySQL离线部署(元数据)
-
mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz
-
jdk-8u241-linux-x64.tar.gz
-
mysql-connector-java-5.1.49.tar.gz
https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.49.tar.gz
-
-
CM离线部署(tar闭源 cloudera核心产品,server agent主从架构)
- cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz
-
Parcel文件离线部署(hdfs、yarn、hive、hbase、zk)
- CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.16.2/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
- CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.16.2/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1
- manifest.json
http://archive.cloudera.com/cdh5/parcels/5.16.2/manifest.json
节点初始化
hosts文件
10.69.24.123 cdh01
10.69.24.124 cdh02
10.69.24.125 cdh03
防火墙
关闭防火墙
systemctl stop firewalld
设置开机禁止启动
systemctl disable firewalld
selinux
vim /etc/selinux/config
将SELINUX=Enforcing改为SELINUX=disable
时区和时钟同步
时区
timedatectl set-timezone Asia/Shanghai
ntp服务器同步
yum install -y ntp
选取cdh01作为时间同步的主节点,02,03作为从节点
vim /etc/ntp.conf
亚洲的时区同步
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
不能访问外网时,允许访问本地时间
server 127.127.1.0 iburst local clock
fudge 127.0.0.1 stratum 10
允许哪个网段的地址能进行同步
restrict 10.69.24.120 mask 255.255.255.248 nomodify notrap
启动ntp服务
systemctl start ntpd
查看ntp服务状态
systemctl status ntpd
使用ntpq -p 查看网络中的NTP服务器
分别在cdh02和cdh03服务器上同步cdh01的时间
/usr/sbin/ntpdate cdh01
设置每隔一段时间进行自动同步,防止时间过长,时间差的不断累积导致不断增大
crontab -e
00 00 * * * /usr/sbin/ntpdate cdh01
# 设置每天00:00与cdh01进行时区同步
crontab文件的一些例子
30 21 * * * /etc/init.d/nginx restart //每晚的21:30重启 nginx。
45 4 1,10,22 * * /etc/init.d/nginx restart //每月1、 10、22日的4 : 45重启nginx。
10 1 * * 6,0 /etc/init.d/nginx restart //每周六、周日的1 : 10重启nginx。
0,30 18-23 * * * /etc/init.d/nginx restart //每天18 : 00至23 : 00之间每隔30分钟重启nginx。
0 23 * * 6 /etc/init.d/nginx restart //每星期六的11 : 00 pm重启nginx。
* */1 * * * /etc/init.d/nginx restart //每一小时重启nginx
* 23-7/1 * * * /etc/init.d/nginx restart //晚上11点到早上7点之间,每 隔一小时重启nginx
0 11 4 * mon-wed /etc/init.d/nginx restart //每月的4号与每周一到周三 的11点重启nginx
0 4 1 jan * /etc/init.d/nginx restart //一月一号的4点重启nginx
*/30 * * * * /usr/sbin/ntpdate 210.72.145.20 //每半小时同步一下时间
jdk
解压JDK到/usr/java并配置环境变量
tar -zxvf jdk-8u241-linux-x64.tar.gz -C /usr/java/
将解压的分发到其他两台服务器
scp java cdh02@root:/usr/
scp java cdh03@root:/usr/
修改权限组为root
chown -R root:root jdk1.8.0_241/
配置环境变量
vi /etc/profile
#JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_241
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
配置文件生效
source /etc/profile
部署MySQL
[root@cdh01 cdh5.16.2]# tar -zxvf mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz -C /usr/local/
[root@cdh01 local]# mv mysql-5.7.31-linux-glibc2.12-x86_64/ mysql
[root@cdh01 local]# mkdir mysql/arch mysql/data mysql/tmp
[root@cdh01 local]# vi /etc/my.cnf
删除所有my.cnf中的配置,复制下面的所有
[client]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4
[mysqld]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
skip-slave-start
skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M
table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600
# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32
#isolation level and default engine
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED
server-id = 1739
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
pid-file = /usr/local/mysql/data/hostname.pid
#open performance schema
log-warnings
sysdate-is-now
binlog_format = ROW
log_bin_trust_function_creators=1
log-error = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7
innodb_write_io_threads=16
relay-log = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info
log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF
# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
#other logs
#general_log =1
#general_log_file = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err
#for replication slave
sync_binlog = 500
#for innodb options
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend
innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M
#根据生产需要,调整pool size
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp
innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2
innodb_locks_unsafe_for_binlog=1
#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on
#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1
[mysqldump]
quick
max_allowed_packet = 128M
[mysql]
no-auto-rehash
default-character-set=utf8mb4
[mysqlhotcopy]
interactive-timeout
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
创建用户组及用户
[root@cdh01 local]# groupadd -g 101 dba
[root@cdh01 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
[root@cdh01 local]# id mysqladmin
uid=514(mysqladmin) gid=101(dba) 组=101(dba),0(root)
[root@cdh01 local]# cp /etc/skel/.* /usr/local/mysql
配置环境变量
[root@hadoop39 local]# vi mysql/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH
unset USERNAME
#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1
## end
6.赋权限和用户组,切换用户mysqladmin,安装
[root@hadoop39 local]# chown mysqladmin:dba /etc/my.cnf
[root@hadoop39 local]# chmod 640 /etc/my.cnf
[root@hadoop39 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@hadoop39 local]# chmod -R 755 /usr/local/mysql
7.配置服务及开机自启动
[root@hadoop39 local]# cd /usr/local/mysql
#将服务文件拷贝到init.d下,并重命名为mysql
[root@hadoop39 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql
#赋予可执行权限
[root@hadoop39 mysql]# chmod +x /etc/rc.d/init.d/mysql
#删除服务
[root@hadoop39 mysql]# chkconfig --del mysql
#添加服务
[root@hadoop39 mysql]# chkconfig --add mysql
[root@hadoop39 mysql]# chkconfig --level 345 mysql on
8.安装libaio及安装mysql的初始db
[root@hadoop39 mysql]# yum -y install libaio
[root@hadoop39 mysql]# sudo su - mysqladmin
hadoop39.ruoze:mysqladmin:/usr/local/mysql:> bin/mysqld \
--defaults-file=/etc/my.cnf \
--user=mysqladmin \
--basedir=/usr/local/mysql/ \
--datadir=/usr/local/mysql/data/ \
--initialize
在初始化时如果加上 –initial-insecure,则会创建空密码的 root@localhost 账号,否则会创建带密码的 root@localhost 账号,密码直接写在 log-error 日志文件中
(在5.6版本中是放在 ~/.mysql_secret 文件里,更加隐蔽,不熟悉的话可能会无所适从)
9.查看临时密码
hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>cat hostname.err |grep password
2017-07-22T02:15:29.439671Z 1 [Note] A temporary password is generated for root@localhost: kFCqrXeh2y(0
hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>
10.启动
/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &
11.登录及修改用户密码
hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>mysql -uroot -p'kFCqrXeh2y(0'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> alter user root@localhost identified by '123456';
Query OK, 0 rows affected (0.05 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' ;
Query OK, 0 rows affected, 1 warning (0.02 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
Bye
12.重启
hadoop39.ruoze:mysqladmin:/usr/local/mysql:> service mysql restart
hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>mysql -uroot -pruozedata
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log MySQL Community Server (GPL)
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>
事先创建db用户
mysql> create database cmf default character set utf8;
GRANT ALL PRIVILEGES ON cmf.* TO 'cmf'@'%' IDENTIFIED BY '123456';
mysql> create database amon default character set utf8;
Query OK, 1 row affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>
部署MySQLJDBC
[root@cdh01 mysql-connector-java-5.1.49]# mv mysql-connector-java-5.1.49.jar /usr/share/java/mysql-connector-java.jar
离线部署CM
三台机器都要安装CM
[root@cdh01 cdh5.16.2]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz cdh02:/etc/opt/cdh5.16.2/
[root@cdh01 cdh5.16.2]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz cdh03:/etc/opt/cdh5.16.2/
[root@cdh01 cdh5.16.2]# mkdir /opt/cloudera-manager
[root@cdh01 cdh5.16.2]# tar -zxvf cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz -C /opt/cloudera-manager/
[root@cdh01 cdh5.16.2]# cd /opt/cloudera-manager/cm-5.16.2/etc/cloudera-scm-agent/
[root@cdh01 cloudera-scm-agent]# vi config.ini
修改为主机名称
选举cdh01的CM为server
[root@cdh01 cloudera-scm-agent]# cd /opt/cloudera-manager/cm-5.16.2/etc/cloudera-scm-server/
[root@cdh01 cloudera-scm-server]# vi db.properties
创建默认用户cloudera-scm,三台都要创建
home目录 目录已经存在不需要创建,不需要登陆
[root@cdh01 cloudera-scm-server]# useradd --system --home=/opt/cloudera-manager/cm-5.16.2/run/cloudera-scm-server/ --no-create-home --shell=/bin/false cloudera-scm
[root@cdh01 cloudera-scm-server]# chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager/
部署parcel文件 cdh01
[root@cdh01 cloudera-scm-server]# mkdir -p /opt/cloudera/parcel-repo
[root@cdh01 cdh5.16.2]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel /opt/cloudera/parcel-repo/
[root@cdh01 cdh5.16.2]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha
[root@cdh01 cdh5.16.2]# mv manifest.json /opt/cloudera/parcel-repo/
[root@cdh01 parcel-repo]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/
所有节点创建安装目录并设置用户组权限
[root@cdh01 parcel-repo]# mkdir -p /opt/cloudera/parcels
[root@cdh01 parcel-repo]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/
启动cm服务
[root@cdh01 etc]# cd /opt/cloudera-manager/cm-5.16.2/etc/init.d/
[root@cdh01 init.d]# ll
总用量 32
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 8871 5月 18 2019 cloudera-scm-agent
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 8417 5月 18 2019 cloudera-scm-server
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 4444 5月 18 2019 cloudera-scm-server-db
[root@cdh01 init.d]#
[root@cdh01 init.d]# ./cloudera-scm-server start
分别启动3台的agent
[root@cdh01 init.d]# ./cloudera-scm-agent start
账号密码:admin admin
第一处叹号
[root@cdh01 init.d]# vi /etc/sysctl.conf
第二处叹号
三台虚拟机都需要设置
[root@cdh02 init.d]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@cdh02 init.d]#
[root@cdh02 init.d]# echo never > /sys/kernel/mm/transparent_hugepage/enabled