机器准备
准备3台机器,操作系统CentOS7.2,配置:至少2核8G
这里我用了3台阿里云的主机(建议学习的小伙伴使用新加坡的机器,下载安装包飞快)
一 节点初始化
1.1 设置所有节点的hosts文件
ip地址设置为内网地址
echo "172.21.119.114 hadoop001">> /etc/hosts
echo "172.21.119.112 hadoop002">> /etc/hosts
echo "172.21.119.113 hadoop003">> /etc/hosts
1.2 关闭防火墙和清空规则
systemctl stop firewalld
systemctl disable firewalld
iptables -F
1.3 关闭所有节点的selinux
将SELINUX=enforcing改为SELINUX=disabled 设置后需要重启才能⽣生效
vi /etc/selinux/config
1.4 设置所有节点的时区
设置所有节点为 亚洲上海时区
timedatectl set-timezone Asia/Shanghai
查看是否设置成功
timedatectl status
如果timedatectl
命令不会使用,可以使用帮助命令
timedatectl --help
1.5 时钟同步
这里才用主从同步,就是hadoop001服务器跟互联网时间同步,hadoop002和hadoop003服务器根据这台服务器的时间进行同步
三台机器一起安装ntp
yum install -y ntp
1.5.1 选取hadoop001为ntp主节点
vi /etc/ntp.conf
在ntp.conf文件添加下面两行
#当外部时间不不可⽤用时,可使⽤用本地硬件时间
server 127.127.1.0 iburst local clock
#允许哪些⽹网段的机器器来同步时间,我这里的网段是172.21.119
restrict 172.21.119.0 mask 255.255.255.0 nomodify notrap
开启ntp服务并查看状态
systemctl start ntpd
systemctl status ntpd
1.5.2 从节点同步
设置hadoop002和hadoop003根据主节点hadoop001设置时间同步
配置/etc/ntp.conf,只要把server改成下面一行
# 同步第一台机器的时钟
server hadoop001 iburst
然后启动ntp
systemctl start ntpd
1.5.3 添加定时任务
每天0点0分,从节点去同步主节点时间
hadoop002,hadoop003从节点都添加如下定时任务
crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001
1.6 修改hostname
hostnamectl set-hostname 主机名
bash ###立即生效
1.7 禁用swap
vi /etc/fstab
注释加载swap分区的那行记录
#/dev/mapper/centos-swap swap swap defaults 0 0
重启机器 reboot
二 jdk部署
2.1 jdk版本选择
选择jdk的时候看下官网的推荐,有些版本会有问题
地址:https://docs.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_jdk
jdk下载地址:https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html
下载好之后,上传到hadoop001机器
然后把jdk安装包拷贝到hadoop002,hadoop003
scp jdk-8u181-linux-x64.tar.gz hadoop002:/root
scp jdk-8u181-linux-x64.tar.gz hadoop003:/root
三台机器一起创建java目录(这里java一定要安装下面这个目录,否则后面会出现问题)
mkdir /usr/java
然后三台机器一起解压
tar -zxvf /root/jdk-8u181-linux-x64.tar.gz -C /usr/java
三台机器修改用户和用户组
chown -R root:root /usr/java
三台机器添加java环境变量
先编辑/etc/profile
vi /etc/profile
再添加如下的环境
# java env
export JAVA_HOME=/usr/java/jdk1.8.0_181
export PATH=${JAVA_HOME}/bin:$PATH
然后生效环境变量,查看是否安装好
到这里说明jdk安装ok
三 MySQL 安装
安装MySQL建议使用5.7 经典版本,别搞个5.8 版本,到时候cdh不支持
可以查看官网的支持版本(如果你使用的cdh版本是5.0-5.9,那么MySQL 5.7不支持)
https://docs.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#cdh_cm_supported_db
MySQL下载地址
https://dev.mysql.com/downloads/mysql/5.7.html#downloads
下载好之后上传到hadoop001
1.将mysql解压,并重命名
tar -zxvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz -C /usr/local
mv /usr/local/mysql-5.7.11-linux-glibc2.5-x86_64 /usr/local/mysql
2.创建必要的数据目录
cd /usr/local/mysql
mkdir arch data tmp
3.编辑my.cnf文件
vi /etc/my.cnf
添加如下内容(我们生产的配置文件,生产上只要把pool size改成8G左右就行了)
[client]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4
[mysqld]
port = 3306
socket = /usr/local/mysql/data/mysql.sock
skip-slave-start
skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M
table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600
# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32
#isolation level and default engine
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED
server-id = 1739
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
pid-file = /usr/local/mysql/data/hostname.pid
#open performance schema
log-warnings
sysdate-is-now
binlog_format = ROW
log_bin_trust_function_creators=1
log-error = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7
innodb_write_io_threads=16
relay-log = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info
log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF
# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
#other logs
#general_log =1
#general_log_file = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err
#for replication slave
sync_binlog = 500
#for innodb options
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend
innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M
#根据生产需要,调整pool size
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp
innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2
innodb_locks_unsafe_for_binlog=1
#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on
#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1
[mysqldump]
quick
max_allowed_packet = 128M
[mysql]
no-auto-rehash
default-character-set=utf8mb4
[mysqlhotcopy]
interactive-timeout
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M
4.创建用户和用户组
groupadd -g 101 dba
useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
如果mysqladmin用户已经存在,可以执行下面命令
usermod -u 514 -g dba -G root -d /usr/local/mysql mysqladmin
5.copy 环境变量配置文件至mysqladmin用户的home目录中,为了后面步骤配置个人环境变量
cp /etc/skel/.* /usr/local/mysql
6.赋权和用户组
chown mysqladmin:dba /etc/my.cnf
chmod 640 /etc/my.cnf
chown -R mysqladmin:dba /usr/local/mysql
chmod -R 755 /usr/local/mysql
7.配置环境变量
先切换到mysqladmin用户
su - mysqladmin
编辑环境配置文件(注意这里编辑的是.bashrc)
vi ~/.bashrc
添加如下内容
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH
然后生效配置文件
source /usr/local/mysql/.bashrc
8.配置服务及开机自启动
先切换到root用户,再执行下面命令
cd /usr/local/mysql
#将服务文件拷贝到init.d下,并重命名为mysql
cp support-files/mysql.server /etc/rc.d/init.d/mysql
#赋予可执行权限
chmod +x /etc/rc.d/init.d/mysql
#添加服务
chkconfig --add mysql
9.安装libaio
yum -y install libaio
10.初始化
切换到mysqladmin用户
su - mysqladmin
执行初始化命令
bin/mysqld \
--defaults-file=/etc/my.cnf \
--user=mysqladmin \
--basedir=/usr/local/mysql/ \
--datadir=/usr/local/mysql/data/ \
--initialize
等待几十秒就ok
11.查看临时密码
临时密码在data目录下的hostname.err里,通过下面命令查看
cat hostname.err | grep password
下图中为安装之后的临时密码
12.启动mysql
service mysql start
13.登陆
mysql -uroot -p'u)eUJxf9p*tw'
修改密码
mysql> alter user root@localhost identified by 'password';
Query OK, 0 rows affected (0.05 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected, 1 warning (0.02 sec)
#刷新权限
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
14.重启mysql
service mysql restart
四 元数据+用户+jdbc包
先登陆mysql
mysql -uroot -p'password'
1.创建元数据库
create database cmf default character set utf8;
create database amon default character set utf8;
2.创建数据库授权用户
grant all privileges on cmf.* to 'cmf'@'%' identified by 'ruozedata';
grant all privileges on amon.* to 'amon'@'%' identified by 'ruozedata';
刷新权限
flush privileges;
3.准备jdbc包
自己下载个jdbc驱动包即可(上传到hadoop001)
创建目录(目录不要随便放置)
mkdir -p /usr/share/java
拷贝驱动包(移除版本号)
cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar
五 CM部署
前面都属于我们准备工作,下面开始CM部署
因为CM是一个主从架构,我们在hadoop001安装Server,在每台机器安装Agent
下载地址:https://archive.cloudera.com/cm5/repo-as-tarball/5.16.1/
wget https://archive.cloudera.com/cm5/repo-as-tarball/5.16.1/cm5.16.1-centos7.tar.gz
对tar包解压
tar -zxvf cm5.16.1-centos7.tar.gz
查看rmp包
cd cm/5.16.1/RPMS/x86_64/
我们只需要用图中框框部分的rpm包即可
cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm 是agent和server公用的包
所以安装server时需要
cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
安装agent时需要
cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm
5.1 安装Server
1.先安装daemons
rpm -ivh cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm --nodeps --force
2.然后安装server
rpm -ivh cloudera-manager-server-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm --nodeps --force
3.修改数据库配置文件
进入配置文件目录
cd /etc/cloudera-scm-server
修改配置文件
vi db.properties
按照自己的参数进行设置
4.启动服务
service cloudera-scm-server start
启动之后,查看日志(要等1分钟左右)
tail -F /var/log/cloudera-scm-server/cloudera-scm-server.log
看到7180端口,说明Server安装成功
5.2 安装Agent
Agent要在三台机器都安装,我们先把daemons和agent都拷贝到hadoop002和hadoop003机器
scp cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm hadoop002:/root
scp cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm hadoop003:/root
scp cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm hadoop002:/root
scp cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm hadoop003:/root
下面对三台机器进行安装
1.hadoop001安装agent(hadoop001刚才已经安装过daemons了)
rpm -ivh cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm --nodeps --force
2.hadoop002和hadoop003安装agent
rpm -ivh cloudera-manager-daemons-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm --nodeps --force
rpm -ivh cloudera-manager-agent-5.16.1-1.cm5161.p0.1.el7.x86_64.rpm --nodeps --force
3.三台机器都编辑config.ini
vi /etc/cloudera-scm-agent/config.ini
都指向Server hadoop001
4.三台机器启动agent
service cloudera-scm-agent start
六 CDH下载parcel并搭建http服务
1.在hadoop001安装httpd
yum install -y httpd
2.下载CDH5.16.1
https://archive.cloudera.com/cdh5/parcels/5.16.1/
命令如下
wget https://archive.cloudera.com/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
wget https://archive.cloudera.com/cdh5/parcels/5.16.1/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
wget https://archive.cloudera.com/cdh5/parcels/5.16.1/manifest.json
下载好之后的文件如下图,
3.在/var/www/html下创建cdh5_parcel目录
mkdir /var/www/html/cdh5_parcel
4.把下载好之后的文件移动到cdh5_parcel目录(注意把.sha1改成.sha)
mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel /var/www/html/cdh5_parcel/
mv CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /var/www/html/cdh5_parcel/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parc
el.sha
mv manifest.json /var/www/html/cdh5_parcel/
5.验证hash值
sha1sum CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
cat CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
6.启动httpd
service httpd start
从下图可以看出,我们本地的仓库安装好了
七访问WebUI
地址为server的ip:port,默认账号:admin,密码:admin
选择免费
显示大页面
我们执行下命令(三台机器都执行)
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
选择Custom Services(上来不要一下子全安装了,先安装基本的,不然到时候出问题都不知道哪里出问题)
到这里安装成功
八 CM界面的问题解决
8.1 解决cloudera manager
8.2 解决zookeeper问题
点击All Hosts->hadoop002
下面图提示,我们NTP服务没有启动,因为我们时间是主从同步的,所以这里直接点击Suppress
到这里都ok了,如下图。