选购机器
准备工作
内网
172.18.32.209 hadoop001
172.18.32.207 hadoop002
172.18.32.208 hadoop003
外网
hadoop001 47.112.138.102
hadoop002 47.112.121.29
hadoop003 47.107.113.28
三个节点分别执行如下操作
echo "172.18.32.209 hadoop001" >> /etc/hosts
echo 172.18.32.207 hadoop002 >> /etc/hosts
echo 172.18.32.208 hadoop003 >> /etc/hosts
[root@hadoop001 software]# scp cdh_soft.zip hadoop002:/root/software
[root@hadoop001 software]# scp cdh_soft.zip hadoop003:/root/software
下载源:
cloudera-manager-centos7-cm5.12.0_x86_64.tar.gz
CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel
CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1
关闭防火墙
每个节点都需要关闭防火墙,云主机默认已经关闭
[root@hadoop001 cdh_soft]# systemctl status firewalld
如果不是云主机,执行以下命令
systemctl stop firewalld
systemctl disable firewalld
iptables -F
关闭所有节点的selinux
[root@hadoop001 cdh_soft]# vim /etc/selinux/config
每个节点都需要关闭selinux,云主机默认已经关闭
如果不是云主机,执行以下命令
vim /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled
设置后需要重启才能⽣生效
设置所有节点的时区⼀一致及时钟同步
如果是阿里云主机可以不设置
如果不是,执行如下命令
时区
timedatectl set-timezone Asia/Shanghai
时间
yum install -y ntp
vi /etc/ntp.conf
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org #当外部时间不不可⽤用时,可使⽤用本地硬件时间
server 127.127.1.0 iburst local clock #允许哪些⽹网段的机器器来同步时间
restrict 172.19.7.0 mask 255.255.255.0 nomodify notrap
systemctl start ntpd
systemctl status ntpd
#验证
ntpq -p
下面这些命令,即使是阿里云主机也要执行
其他从节点停⽌止禁⽤用ntpd服务
[root@hadoop002 software]# systemctl stop ntpd
[root@hadoop003 software]# systemctl stop ntpd
[root@hadoop002 software]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop003 software]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop002 software]# /usr/sbin/ntpdate hadoop001
3 Jun 18:27:52 ntpdate[18502]: adjust time server 172.18.32.209 offset -0.054042 sec
[root@hadoop003 software]# /usr/sbin/ntpdate hadoop001
3 Jun 18:28:02 ntpdate[18491]: adjust time server 172.18.32.209 offset -0.051221 sec
[root@hadoop002 software]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001
[root@hadoop003 software]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001
三个节点同时安装jdk
[root@hadoop001 software]# mkdir /usr/java
[root@hadoop002 software]# mkdir /usr/java
[root@hadoop003 software]# mkdir /usr/java
[root@hadoop001 cdh_soft]# tar -xzf jdk-8u45-linux-x64.gz -C /usr/java/
[root@hadoop002 software]# tar -xzf jdk-8u45-linux-x64.gz -C /usr/java/
[root@hadoop003 software]# tar -xzf jdk-8u45-linux-x64.gz -C /usr/java/
[root@hadoop001 cdh_soft]# chown -R root:root /usr/java/jdk1.8.0_45
[root@hadoop002 software]# chown -R root:root /usr/java/jdk1.8.0_45
[root@hadoop003 software]# chown -R root:root /usr/java/jdk1.8.0_45
vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile
which java
hadoop001节点离线部署MySQL5.7
在这里直接使用yum安装MySQL5.7,如果离线部署MySQL5.7,则查看离线部署MySQL5.7
[root@hadoop001 software]# yum -y install mysql57-community-release-el7-10.noarch.rpm
[root@hadoop001 software]# yum -y install mysql-community-server
[root@hadoop001 software]# systemctl start mysqld.service
[root@hadoop001 software]# systemctl enable mysqld.service
[root@hadoop001 software]# grep "password" /var/log/mysqld.log
2019-06-03T11:55:13.867469Z 1 [Note] A temporary password is generated for root@localhost: MiWsE_Oxr9+G
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'new password';
创建CDH的元数据库和⽤用户、amon服务的数据库及⽤用户
mysql> create database cmf DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> create database amon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'Aaron123456#';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'Aaron123456#';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
hadoop001节点部署mysql jdbc jar
[root@hadoop001 software]# mkdir -p /usr/share/java/
[root@hadoop001 cdh_soft]# mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar
[root@hadoop001 cdh_soft]# cp mysql-connector-java.jar /usr/share/java/
离线部署cm server及agent
所有节点创建⽬目录及解压
mkdir /opt/cloudera-manager
tar -zxf cloudera-manager-centos7-cm5.12.0_x86_64.tar.gz -C /opt/cloudera-manager/
所有节点修改agent的配置,指向server的节点hadoop001
sed -i "s/server_host=localhost/server_host=hadoop001/g" /opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-agent/config.ini
主节点修改server的配置:
[root@hadoop001 CDH5.12.0]# vi /opt/cloudera-manager/cm-5.12.0/etc/cloudera-scm-server/db.properties
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#
# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql
# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=hadoop001
# The database name
com.cloudera.cmf.db.name=cmf
# The database user
com.cloudera.cmf.db.user=cmf
# The database user's password
com.cloudera.cmf.db.password=Aaron123456#
# The db setup type
# By default, it is set to INIT
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
#com.cloudera.cmf.db.setupType=INIT
com.cloudera.cmf.db.setupType=EXTERNAL
所有节点创建⽤用户
useradd --system --home=/opt/cloudera-manager/cm-5.12.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
⽬目录修改⽤用户及⽤用户组
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager
hadoop001节点部署离线parcel源
[root@hadoop001 CDH5.12.0]# mkdir -p /opt/cloudera/parcel-repo
[root@hadoop001 CDH5.12.0]# cp CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel /opt/cloudera/parcel-repo/
[root@hadoop001 CDH5.12.0]# cp CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.12.0-1.cdh5.12.0.p0.29-el7.parcel.sha
[root@hadoop001 CDH5.12.0]# cp manifest.json /opt/cloudera/parcel-repo/
[root@hadoop001 CDH5.12.0]# chown -R cloudera-scm:cloudera-scm /opt/cloudera/
所有节点创建软件安装⽬目录、⽤用户及⽤用户组权限
mkdir -p /opt/cloudera/parcels
chown -R cloudera-scm:cloudera-scm /opt/cloudera/
hadoop001节点启动Server
[root@hadoop001 CDH5.12.0]# /opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-server start
Starting cloudera-scm-server: [ 确定 ]
阿⾥里里云web界⾯面,设置该hadoop001节点防⽕火墙放开7180端⼝
等待1min,打开 http://hadoop001:7180 账号密码:admin/admin
假如打不不开,去看server的log,根据错误仔细排查错误
所有节点启动Agent
/opt/cloudera-manager/cm-5.12.0/etc/init.d/cloudera-scm-agent start
三个主机同时运行下面三条命令
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
建议将/proc/sys/vm/swappiness设置为最⼤大值10。 swappiness值控制操作系统尝试交换内存的积极; swappiness=0:表示最⼤大限度使⽤用物理理内存,之后才是swap空间; swappiness=100:表示积极使⽤用swap分区,并且把内存上的数据及时搬迁到swap空间; 如果是混合服务器器,不不建议完全禁⽤用swap,可以尝试降低swappiness。
临时调整:
sysctl vm.swappiness=10
永久调整:
cat << EOF >> /etc/sysctl.conf
# Adjust swappiness value
vm.swappiness=10
EOF
已启⽤用透明⼤大⻚页⾯面压缩,可能会导致重⼤大性能问题,建议禁⽤用此设置。 临时调整:
echo never > /sys/kernel/mm/transparent_hugepage/defrag echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久调整:
cat << EOF >> /etc/rc.d/rc.local
# Disable transparent_hugepage
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
EOF
# centos7.x系统,需要为"/etc/rc.d/rc.local"文件赋予执⾏行行权限 chmod +x /etc/rc.d/rc.local
⾃自定义服务,选择部署Zookeeper、HDFS、Yarn服务