CentOS7.6 在线安装 CDH6.3.0

准备工作

准备服务器

准备4台centos8 一台master 3台slave

配置网络

启用网卡

centos7最小化安装 无ifconfig 启用网卡.

• 修改hostname

sudo hostnamectl set-hostname vic01
sudo hostnamectl

• 修改hosts

sudo vim /etc/hosts

将以下内容添加到hosts里

10.2.0.10 vic0 vic0
10.2.0.11 vic1 vic1
10.2.0.12 vic2 vic2
10.2.0.13 vic3 vic3

配置集群SSH免密登录

scp免密码拷贝和ssh免密码登录
.
在主节点上执行ssh-keygen -t rsa一路回车，生成无密码的密钥对。

将公钥添加到认证文件中：

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

并设置authorized_keys的访问权限

chmod 600 ~/.ssh/authorized_keys

scp文件到所有datenode节点：

scp ~/.ssh/authorized_keys root@vic1:~/.ssh
scp ~/.ssh/authorized_keys root@vic2:~/.ssh
scp ~/.ssh/authorized_keys root@vic3:~/.ssh

PS ：shell 分发

同时配置多台机器可设置 Xshell 脚本分发

时间同步配置

yum -y install ntp
systemctl start ntpd 
systemctl enable ntpd 

在vic0节点

vi /etc/ntp.conf

server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 1.cn.pool.ntp.org nomodify notrap noquery
restrict 2.cn.pool.ntp.org nomodify notrap noquery
restrict 3.cn.pool.ntp.org nomodify notrap noquery
server 127.127.1.0
fudge 127.127.1.0 stratum 10

systemctl start ntpd.service
systemctl enable ntpd.service
ntpstat

在datanode节点

vim /etc/ntp.conf

server vic0 prefer

systemctl start ntpd.service
systemctl enable ntpd.service
ntpdate -u vic0
ntpstat

配置CDH Repository

导入repo

sudo curl -o /etc/yum.repos.d/cloudera-manager.repo https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/cloudera-manager.repo
sudo rpm --import https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPM-GPG-KEY-cloudera 

安装依赖

sudo yum install -y bind-utils psmisc libxslt cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs httpd mod_ssl openssl-devel python-psycopg2 MySQL-python /lib/lsb/init-functions libpq.so.5

wget http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster/5.5.37-25.10/RPM/rhel6/x86_64/Percona-XtraDB-Cluster-shared-55-5.5.37-25.10.756.el6.x86_64.rpm
sudo rpm -ivh Percona-XtraDB-Cluster-shared-55-5.5.37-25.10.756.el6.x86_64.rpm

安装Oracle JDK

centos7.6 安装配置Oracle JDK

分发JDK

scp -r /usr/java/jdk1.8.0_271 root@vic1:/usr/java/
scp -r /usr/java/jdk1.8.0_271 root@vic2:/usr/java/
scp -r /usr/java/jdk1.8.0_271 root@vic3:/usr/java/

安装数据库

centos7.6 安装配置 mysql5.7

配置mysql

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 16M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

安装mysql 驱动

下载解压驱动

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

一定定要改名mysql-connector-java.jar 放到/usr/share/java路径

进入数据库创建数据库

mysql -u root -p

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

设置CDH的Database

创建数据库账号

创建cdh6给cdh使用

create user chd6@% indentified by '123456'

给账号赋权

GRANT ALL ON scm.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON amon.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON rman.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON hue.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON metastore.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON sentry.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON nav.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON navms.* TO 'cdh6'@'%' IDENTIFIED BY '123456';
GRANT ALL ON oozie.* TO 'cdh6'@'%' IDENTIFIED BY '123456';

namenode安装

sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

systemctl start cloudera-scm-server 

查看日志

sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

官方推荐Enable Auto-TLS（配置复杂不建议启用）

sudo JAVA_HOME=/usr/java/jdk1.8.0_271 /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services

设置CDH的Database

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm cdh6

重启

systemctl restart cloudera-scm-server 

地址：http://vic0:7180
用户：admin
密码：admin

datanode安装

sudo yum install cloudera-manager-daemons cloudera-manager-agent

vi /etc/cloudera-scm-agent/config.ini

server_host=vic0

启动agent

systemctl start cloudera-scm-agent