文章目录
CDH 6.3.2安装回顾
花了一天的时间安装了CDH6.3.2
总结一下以便于之后维护和学习
这个博客把如何安装虚拟机写得很好,推荐看一下:
环境篇:VMware Workstation安装Centos7
一、准备第一台虚拟机
买的是服务器。服务器装的是EXSI系统。
虚拟出三台机器。
分别为
hostname | cpu | 内存 | 硬盘 |
---|---|---|---|
cdh201 | 24核 | 24G | 600G |
cdh202 | 12核 | 24G | 600G |
cdh203 | 12核 | 16G | 600G |
然后开始虚拟出第一台虚拟机。
在最开始装的时候,
-
设置分区
- boot给1G swap查了下一般是内存的2倍 都给了40G 其余的全给根目录
- 这里面有好几种格式btrfs 还有LVM 还有 标准分区 我用了标准分区
- 类型有好几种ext4 等等 我用了默认那个 忘了是啥了
-
设置静态ip
- 我直接设置成了192.168.50.201
-
关闭kdump
-
设置root 然后密码 我的是123456
-
开机
-
查看ip是否正确 (是否是静态的,最好设置为静态的)
-
ifconfig
-
su root
切换到root用户 -
vim /etc/sysconfig/network-scripts/ifcfg-ens33
BOOTPROTO=“static” IPADDR="192.168.50.201" NETMASK="255.255.255.0" GATEWAY="192.168.50.1" DNS1="119.29.29.29" ####################### 解释一下上面几个参数 第零个:设置为静态IP 第一个:你自己设置的静态IP地址 只要跟之前的不冲突不重复就可以 第二个:设置的子网掩码 第三个:网关 我这里是写的路由器的地址 第四个:这个是腾讯的DNS,我个人比较喜欢腾讯哈哈哈
-
搞完之后看看能不能
ping
通www.baidu.com
(百度真惨,天天被拿来ping
网络是否畅通) -
这里我们提前配置一下hosts文件
vim /etc/hosts
vim /etc/hosts #增加如下内容: 192.168.50.201 cdh201 192.168.50.202 cdh202 192.168.50.203 cdh203
-
-
yum -y update
-
yum -y install httpd ntp chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl openssl-devel python-psycopg2 MySQL-python python-devel telnet pcre-devel gcc gcc-c++
(待补充) -
关闭防火墙
firewalld
systemctl stop firewalld.service systemctl disable firewalld.service systemctl status firewalld.service
-
永久关闭网络管理让机子强行执行静态分配
#停止网络管理 systemctl stop NetworkManager #删除网络管理 systemctl disable NetworkManager #重启网络 systemctl restart network.service
-
关闭
iptables
systemctl stop iptables.service systemctl disable iptables.service systemctl status iptables.service
-
禁用
selinux
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config setenforce 0 getenforce sestatus ************或者用下面这种vim方式*********** #临时关闭内核防火墙 setenforce 0 getenforce # 永久关闭内核防火墙 vim /etc/selinux/config
-
设置
ntp
同步-
按下面步骤
#更新yum源ntp yum -y install ntp #查询机器时间 date #时间同步 用阿里云的ntp ntpdate ntp1.aliyun.com #查看时间同步服务 service ntpd status #临时启用时间同步服务 service ntpd start #配置ntpd永久生效(重启生效) chkconfig ntpd on
-
-
装
jdk
-
先看看我们的机器上是否已经有jdk了,我昨天看我这上面有N多jdk
-
rpm -qa | grep java
或者rpm -qa | grep jdk
-
如果有 就执行
rpm -e + 软件包的名字
-
如果提示有依赖,要么删去依赖的软件,要么直接用
rpm -e --nodeps + 软件包
-
反正删完就行。
-
找个地方放我们下载的CDH官方给的JDK
-
我们先装一下jdk
-
我装到了
/opt/data
中mkdir /opt/data 把oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm进行上传 rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm rm -rf oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm vim /etc/profile #添加 export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar source /etc/profile java -version
-
-
java 装完 可以克隆虚拟机
二、克隆另外两台从机
-
第一步要先复制文件,注册
-
然后开机
-
开机后第一件事需要改
hostname
-
vim /etc/hostname
然后直接在里面编写cdh202
-
vim /etc/sysconfig/network
#增加如下内容: NETWORKING=yes HOSTNAME=cdh202
-
接着到
vim /etc/sysconfig/network-scripts/ifcfg-ens33
中 配置
BOOTPROTO=“static”
IPADDR="192.168.50.202"
NETMASK="255.255.255.0"
GATEWAY="192.168.50.1"
DNS1="119.29.29.29"
-
因为上面已经配了hosts 并且克隆 这里就不用弄了
-
从节点配置NTP
使用crontab定时任务 crontab -e 添加定时任务(每分钟和主机同步)内容如下: 0-59/1 * * * * /usr/sbin/ntpdate cdh201
-
配置三台之间的SSH
- 在所有节点上执行
ssh-keygen -t rsa
- 在所有节点上执行
ssh-copy-id cdh201 ssh-copy-id cdh202 ssh-copy-id cdh203
- 3 * 3 = 9 我们共计输入9遍密码
- 在所有节点上执行
三、开始安装CM
-
首先我们搞一个本地的yum源
#更新yum源httpd yum -y install httpd #查看httpd状态 systemctl status httpd.service #启动httpd service httpd start #配置httpd永久生效(重启生效) chkconfig httpd on ****************** #更新yum源yum-utils createrepo yum -y install yum-utils createrepo #进入yum源路径 cd /var/www/html/ #创建cm文件夹 mkdir cm
-
把这个链接中的所有东西放到cm文件夹中。并执行
createrepo /var/www/html/cm/
-
创建本地repo文件(这里我之前改了名字,把自己坑了。记住就写
cloudera-manager.repo
)vim /etc/yum.repos.d/cloudera-manager.repo [cloudera-manager] name=Cloudera Manager, Version yum baseurl=http://192.168.50.201/cm gpgcheck=0 enabled=1
-
分发
scp -r /etc/yum.repos.d/cloudera-manager.repo root@cdh202:/etc/yum.repos.d/ scp -r /etc/yum.repos.d/cloudera-manager.repo root@cdh203:/etc/yum.repos.d/
-
搞完上面的 我们在每个节点上都开始执行
yum clean all yum makecache # 验证 yum list | grep cloudera-manager # 或者浏览器输入 http://192.168.50.201/cm # 进行访问
四、主节点装MySQL
(此处采用了官方文档的安装方法,既简单又省事,比用别的方法好太多了)
①
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
②
sudo systemctl stop mysqld
③
Move old InnoDB log files
把下面这两个文件
/var/lib/mysql/ib_logfile0
and
/var/lib/mysql/ib_logfile1
out of /var/lib/mysql/
移出这个文件夹(随便放哪儿都行)
to a backup location.
④
vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
⑤
sudo systemctl enable mysqld
sudo systemctl start mysqld
sudo /usr/bin/mysql_secure_installation
[...]
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!
⑥
这一步需要下载jdbc的jar包放入/usr/share/java/
目录下并改名为mysql-connector-java.jar
Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html (in .tar.gz format). As of the time of writing, you can download version 5.1.46 using wget as follows:
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
Extract the JDBC driver JAR file from the downloaded file. For example:
tar zxvf mysql-connector-java-5.1.46.tar.gz
Copy the JDBC driver, renamed, to /usr/share/java/. If the target directory does not yet exist, create it. For example:
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
#进入mysql(输入修改后的密码)
mysql -u root -p
#scm库和权限暂时不创建,后面指定数据库,会自动创建
#CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
#GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
#####注意此处再授权一个本主机名地址,不然web页面配置很容易出错,注意修改本地主机名hostname
GRANT ALL ON amon.* TO 'amon'@'本主机名' IDENTIFIED BY 'amon';
#刷新源
FLUSH PRIVILEGES;
#检查权限是否正确
show grants for 'amon'@'%';
show grants for 'rman'@'%';
show grants for 'hive'@'%';
show grants for 'hue'@'%';
show grants for 'oozie'@'%';
#退出
quit
#重启服务
service mysql restart
五、通过yum安装daemons,agent,server
主节点
yum list | grep cloudera-manager
yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
从节点
yum list | grep cloudera-manager
yum -y install cloudera-manager-daemons cloudera-manager-agent
六、配置本地Parcel存储库
将CDH下载的文件,上传到指定文件夹(主节点)
cd /opt/cloudera/parcel-repo/
七、scm数据库准备工作
1) 执行脚本scm_prepare_database.sh
#设置Cloudera Manager 数据库
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -uroot -p'123456' scm scm scm
#进入mysql(这里是上面改的密码root)
mysql -uroot -p123456
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
FLUSH PRIVILEGES;
show grants for 'scm'@'%';
quit
2) 停止ClouderaManager服务
service cloudera-scm-server stop
service cloudera-scm-server-db stop
3) 删除内嵌的默认数据库PostgreSQL的配置
rm -f /etc/cloudera-scm-server/db.mgmt.properties
4) 启动ClouderaManager服务
service cloudera-scm-server start
这时候就能网页登陆7180查看集群情况了。
八、安装
九、Spark/Flink/Phoenix等组件的集成
cdh6.3.2默认安装的是spark2.4版本
Flink需要到Flink的jar包和安装包下载地址下载
需要把jar包放到csd文件夹 parcel包放到parcel文件夹下 注意要把mainfest.json放进去。
Phoenix和上面的类似
phoenix6.2.0版本下载地址
To be continued(2020年8月11日)