CentOs安装CDH5.16.2(离线部署)

CDH5.16.2真正的离线部署

参考视频:若泽大数据

  • MySQL离线部署(元数据)

    • mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz

    • jdk-8u241-linux-x64.tar.gz

    • mysql-connector-java-5.1.49.tar.gz

    https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.49.tar.gz
    
  • CM离线部署(tar闭源 cloudera核心产品,server agent主从架构)

    • cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz
    http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz
    
  • Parcel文件离线部署(hdfs、yarn、hive、hbase、zk)

    • CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
    http://archive.cloudera.com/cdh5/parcels/5.16.2/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel
    
    • CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1
    http://archive.cloudera.com/cdh5/parcels/5.16.2/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1
    
    • manifest.json
    http://archive.cloudera.com/cdh5/parcels/5.16.2/manifest.json
    

    节点初始化

    hosts文件

    10.69.24.123 cdh01

    10.69.24.124 cdh02

    10.69.24.125 cdh03

    防火墙

    关闭防火墙

    systemctl stop firewalld
    

    设置开机禁止启动

    systemctl disable firewalld
    

    selinux

    vim /etc/selinux/config
    

    将SELINUX=Enforcing改为SELINUX=disable

    时区和时钟同步

    时区

    timedatectl set-timezone Asia/Shanghai
    

    ntp服务器同步

    yum install -y ntp
    

    选取cdh01作为时间同步的主节点,02,03作为从节点

    vim /etc/ntp.conf 
    

在这里插入图片描述
亚洲的时区同步

server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org

不能访问外网时,允许访问本地时间

server 127.127.1.0 iburst local clock
fudge 127.0.0.1 stratum 10

允许哪个网段的地址能进行同步

restrict 10.69.24.120 mask 255.255.255.248 nomodify notrap

启动ntp服务

systemctl start ntpd

查看ntp服务状态

systemctl status ntpd

在这里插入图片描述
使用ntpq -p 查看网络中的NTP服务器
在这里插入图片描述
分别在cdh02和cdh03服务器上同步cdh01的时间

/usr/sbin/ntpdate cdh01

在这里插入图片描述
设置每隔一段时间进行自动同步,防止时间过长,时间差的不断累积导致不断增大

crontab -e

在这里插入图片描述
00 00 * * * /usr/sbin/ntpdate cdh01 # 设置每天00:00与cdh01进行时区同步
crontab文件的一些例子

30 21 * * * /etc/init.d/nginx restart             //每晚的21:30重启 nginx。
45 4 1,10,22 * * /etc/init.d/nginx restart        //每月1、 10、22日的4 : 45重启nginx。
10 1 * * 6,0 /etc/init.d/nginx restart            //每周六、周日的1 : 10重启nginx。
0,30 18-23 * * * /etc/init.d/nginx restart        //每天18 : 00至23 : 00之间每隔30分钟重启nginx。
0 23 * * 6 /etc/init.d/nginx restart              //每星期六的11 : 00 pm重启nginx。
* */1 * * * /etc/init.d/nginx restart             //每一小时重启nginx
* 23-7/1 * * * /etc/init.d/nginx restart          //晚上11点到早上7点之间,每 隔一小时重启nginx
0 11 4 * mon-wed /etc/init.d/nginx restart        //每月的4号与每周一到周三 的11点重启nginx
0 4 1 jan * /etc/init.d/nginx restart             //一月一号的4点重启nginx
*/30 * * * * /usr/sbin/ntpdate 210.72.145.20      //每半小时同步一下时间

jdk

解压JDK到/usr/java并配置环境变量

tar -zxvf jdk-8u241-linux-x64.tar.gz -C /usr/java/

将解压的分发到其他两台服务器

scp java cdh02@root:/usr/
scp java cdh03@root:/usr/

修改权限组为root

chown -R root:root jdk1.8.0_241/

配置环境变量

vi /etc/profile
#JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_241
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

配置文件生效

source /etc/profile

部署MySQL

[root@cdh01 cdh5.16.2]# tar -zxvf mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz -C /usr/local/
[root@cdh01 local]# mv mysql-5.7.31-linux-glibc2.12-x86_64/ mysql
[root@cdh01 local]# mkdir mysql/arch mysql/data mysql/tmp
[root@cdh01 local]# vi /etc/my.cnf
删除所有my.cnf中的配置,复制下面的所有
[client]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4

[mysqld]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock

skip-slave-start

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600

# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32 

#isolation level and default engine 
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id  = 1739
basedir     = /usr/local/mysql
datadir     = /usr/local/mysql/data
pid-file     = /usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format = ROW
log_bin_trust_function_creators=1
log-error  = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7

innodb_write_io_threads=16

relay-log  = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info

log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF

# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON

#other logs
#general_log =1
#general_log_file  = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
sync_binlog = 500


#for innodb options 
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M

#根据生产需要,调整pool size 
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp

innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2

innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
default-character-set=utf8mb4

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

创建用户组及用户
[root@cdh01 local]# groupadd -g 101 dba
[root@cdh01 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
[root@cdh01 local]# id mysqladmin
uid=514(mysqladmin) gid=101(dba) 组=101(dba),0(root)
[root@cdh01 local]# cp /etc/skel/.* /usr/local/mysql
配置环境变量
[root@hadoop39 local]# vi mysql/.bash_profile
# .bash_profile
# Get the aliases and functions

if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH


unset USERNAME

#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1

## end

6.赋权限和用户组,切换用户mysqladmin,安装
[root@hadoop39 local]# chown  mysqladmin:dba /etc/my.cnf 
[root@hadoop39 local]# chmod  640 /etc/my.cnf  


[root@hadoop39 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@hadoop39 local]# chmod -R 755 /usr/local/mysql 


7.配置服务及开机自启动
[root@hadoop39 local]# cd /usr/local/mysql
#将服务文件拷贝到init.d下,并重命名为mysql
[root@hadoop39 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql 
#赋予可执行权限
[root@hadoop39 mysql]# chmod +x /etc/rc.d/init.d/mysql
#删除服务
[root@hadoop39 mysql]# chkconfig --del mysql
#添加服务
[root@hadoop39 mysql]# chkconfig --add mysql
[root@hadoop39 mysql]# chkconfig --level 345 mysql on

8.安装libaio及安装mysql的初始db
[root@hadoop39 mysql]# yum -y install libaio
[root@hadoop39 mysql]# sudo su - mysqladmin

hadoop39.ruoze:mysqladmin:/usr/local/mysql:> bin/mysqld \
--defaults-file=/etc/my.cnf \
--user=mysqladmin \
--basedir=/usr/local/mysql/ \
--datadir=/usr/local/mysql/data/ \
--initialize

在初始化时如果加上 –initial-insecure,则会创建空密码的 root@localhost 账号,否则会创建带密码的 root@localhost 账号,密码直接写在 log-error 日志文件中
(在5.6版本中是放在 ~/.mysql_secret 文件里,更加隐蔽,不熟悉的话可能会无所适从)

9.查看临时密码
hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>cat hostname.err |grep password 
2017-07-22T02:15:29.439671Z 1 [Note] A temporary password is generated for root@localhost: kFCqrXeh2y(0
hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>


10.启动
/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &

11.登录及修改用户密码
hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>mysql -uroot -p'kFCqrXeh2y(0'
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> alter user root@localhost identified by '123456';
Query OK, 0 rows affected (0.05 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' ;
Query OK, 0 rows affected, 1 warning (0.02 sec)


mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> exit;
Bye

12.重启
hadoop39.ruoze:mysqladmin:/usr/local/mysql:> service mysql restart

hadoop39.ruoze:mysqladmin:/usr/local/mysql/data:>mysql -uroot -pruozedata
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log MySQL Community Server (GPL)

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

事先创建db用户
mysql> create database cmf default character set utf8;
GRANT ALL PRIVILEGES ON cmf.* TO 'cmf'@'%' IDENTIFIED BY '123456';
mysql> create database amon default character set utf8;   
Query OK, 1 row affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456';      
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> 
部署MySQLJDBC
[root@cdh01 mysql-connector-java-5.1.49]# mv mysql-connector-java-5.1.49.jar /usr/share/java/mysql-connector-java.jar

离线部署CM

三台机器都要安装CM

[root@cdh01 cdh5.16.2]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz cdh02:/etc/opt/cdh5.16.2/
[root@cdh01 cdh5.16.2]# scp cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz cdh03:/etc/opt/cdh5.16.2/
[root@cdh01 cdh5.16.2]# mkdir /opt/cloudera-manager
[root@cdh01 cdh5.16.2]# tar -zxvf cloudera-manager-centos7-cm5.16.2_x86_64.tar.gz -C /opt/cloudera-manager/
[root@cdh01 cdh5.16.2]# cd /opt/cloudera-manager/cm-5.16.2/etc/cloudera-scm-agent/
[root@cdh01 cloudera-scm-agent]# vi config.ini 

修改为主机名称
在这里插入图片描述
选举cdh01的CM为server

[root@cdh01 cloudera-scm-agent]# cd /opt/cloudera-manager/cm-5.16.2/etc/cloudera-scm-server/
[root@cdh01 cloudera-scm-server]# vi db.properties 

在这里插入图片描述

创建默认用户cloudera-scm,三台都要创建
home目录 目录已经存在不需要创建,不需要登陆

[root@cdh01 cloudera-scm-server]# useradd --system --home=/opt/cloudera-manager/cm-5.16.2/run/cloudera-scm-server/ --no-create-home --shell=/bin/false cloudera-scm
[root@cdh01 cloudera-scm-server]# chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager/

部署parcel文件 cdh01

[root@cdh01 cloudera-scm-server]# mkdir -p /opt/cloudera/parcel-repo
[root@cdh01 cdh5.16.2]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel /opt/cloudera/parcel-repo/
[root@cdh01 cdh5.16.2]# mv CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.2-1.cdh5.16.2.p0.8-el7.parcel.sha
[root@cdh01 cdh5.16.2]# mv manifest.json /opt/cloudera/parcel-repo/
[root@cdh01 parcel-repo]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/

所有节点创建安装目录并设置用户组权限
[root@cdh01 parcel-repo]# mkdir -p /opt/cloudera/parcels
[root@cdh01 parcel-repo]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/

启动cm服务

[root@cdh01 etc]# cd /opt/cloudera-manager/cm-5.16.2/etc/init.d/
[root@cdh01 init.d]# ll
总用量 32
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 8871 5月  18 2019 cloudera-scm-agent
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 8417 5月  18 2019 cloudera-scm-server
-rwxr-xr-x. 1 cloudera-scm cloudera-scm 4444 5月  18 2019 cloudera-scm-server-db
[root@cdh01 init.d]# 
[root@cdh01 init.d]# ./cloudera-scm-server start
分别启动3台的agent
[root@cdh01 init.d]# ./cloudera-scm-agent start

账号密码:admin admin
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
第一处叹号

[root@cdh01 init.d]# vi /etc/sysctl.conf  

在这里插入图片描述
第二处叹号
三台虚拟机都需要设置

[root@cdh02 init.d]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@cdh02 init.d]# 
[root@cdh02 init.d]# echo never > /sys/kernel/mm/transparent_hugepage/enabled

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

  • 3
    点赞
  • 8
    收藏
    觉得还不错? 一键收藏
  • 1
    评论
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值