CDH6.2.0部署

最新推荐文章于 2024-04-14 16:28:52 发布

沪上求之

最新推荐文章于 2024-04-14 16:28:52 发布

阅读量1.8k

点赞数

文章标签： mysql 数据库 database 大数据 hadoop

本文链接：https://blog.csdn.net/fzxiaocai/article/details/123430531

版权

CDH6.2.0部署

服务器信息

服务器	主机名	安装服务	密码等信息
192.168.66.10	mysql.cdh.lczy.com	cloudera-manager-server httpd mysql-5.7	数据库初始密码和安装路径信息如下: mysql_root_pwd=“CDH6.2_lczy” mysql_install_path="/usr/local"
192.168.66.11	cdh11.lczy.com	简单初始化 jdk
192.168.66.12	cdh12.lczy.com	简单初始化 jdk
192.168.66.13	cdh13.lczy.com	简单初始化 jdk

1.思路

服务器需要初始化一些信息，比如防火墙，时间，主机名，免密配置等
cdh资源，为了安装方便需要提前下载到本地。考虑到后续新增节点方便，所以配置一个本地文件服务器，方便下载使用。
安装数据库，备份
安装scm 然后启动进行可视化交互界面
最主要的简要介绍下cdh版本

涉及端口
CDH组件版本
https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_62_packaging.html
文档地址
https://docs.cloudera.com/documentation/enterprise/6/6.2/topics/cm_ig_ports_cm.html
文件下载地址–
https://archive.cloudera.com/cdh6/6.2.0/parcels

现在需要订阅了–也可以通过百度云分享下载
https://download.csdn.net/download/fzxiaocai/83750080?spm=1001.2014.3001.5503

2.初始化–每台都做

主机名

	 /usr/bin/hostnamectl set-hostname "${hostname}"   # "${hostname}" 替换成任意主机名即可  
    echo "HOSTNAME=${hostname}" >/etc/sysconfig/network

yum安装依赖工具

yum install -y wget vim net-tools unzip ntpdate lrzsz expect

centos优化配置

  	echo never >/sys/kernel/mm/transparent_hugepage/defrag
    echo 0 >/proc/sys/vm/swappiness
    echo never >/sys/kernel/mm/transparent_hugepage/enabled

防火墙配置

    sed -i "s#SELINUX=enforcing#SELINUX=disabled#g" /etc/selinux/config
    setenforce 0
    systemctl stop firewalld
    systemctl disable firewalld

关闭不必要服务

    systemctl stop postfix
    systemctl disable postfix

时间同步配置
ntp_server=ntp.aliyun.com

	timedatectl set-timezone 'Asia/Shanghai' &>/dev/null   #时区
    rm -rf /etc/localtime
    ln -s /usr/share/zoneinfo/Asia/Shanghai /etc/localtime &>/dev/null
	ntpdate -u "${ntp_server}"  #同步
    echo "0 1 * * * /usr/sbin/ntpdate -u ${ntp_server}" >crontab_conf && crontab crontab_conf && rm -f crontab_conf  ##定时任务同步

jdk安装

    yum localinstall -y oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm &>/dev/null
    ln -s /usr/java/jdk1.8.0_181-cloudera /usr/java/jdk1.8

3.文件准备+配置–仅192.168.66.10

##提前准备好文件--CDH资源
CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel 
CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1 
manifest.json 
cloudera-manager.repo 
RPM-GPG-KEY-cloudera 
allkeys.asc 
cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm 
cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm 
cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm 
cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm 
enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm

yum install -y httpd createrepo
sed -i '145i\IndexOptions NameWidth=100 Charset=UTF-8 FancyIndexing FoldersFirst' /etc/httpd/conf/httpd.conf
 ## FancyIndexing 开启目录浏览修饰   FoldersFirst 目录优先排在前面
 ## NameWidth=85 表示文件名可以最多显示85个英文字符   Charset=UTF-8 字符集

##文件归位--上传文件到指定目录
[root@mysql ~]# cd  /var/www/html
[root@mysql html]# tree .
.
└── cloudera-repos
    ├── cdh6
    │   └── 6.2.0
    │       └── parcels
    │           ├── CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
    │           ├── CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1
    │           └── manifest.json
    └── cm6
        └── 6.2.0
            ├── allkeys.asc
            └── redhat7
                └── yum
                    ├── cloudera-manager.repo
                    ├── RPM-GPG-KEY-cloudera
                    └── RPMS
                        └── x86_64
                            ├── cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
                            ├── cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
                            ├── cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
                            ├── cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm
                            ├── enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm
                            └── oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm


/var/www/html/cloudera-repos/cdh6/6.2.0/parcels
/var/www/html/cloudera-repos/cm6/6.2.0/redhat7/yum/RPMS/x86_64
systemctl restart httpd

cd /var/www/html/cloudera-repos/cm6/6.2.0/redhat7/yum
createrepo .
wget http://192.168.66.10/cloudera-repos/cm6/6.2.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/  ## 换成自己的地址，或者直接copy过去也行
sed -i "s|https://archive.cloudera.com|http://192.168.66.10/cloudera-repos|g" /etc/yum.repos.d/cloudera-manager.repo  ##替换yum安装CDH的地址，不从官网拉，直接拉去本地http

sudo rpm --import http://192.168.66.10/cloudera-repos/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera &>/dev/null   ## 换成自己的地址，或者直接 /var/www/html/cloudera-repos/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera 也行
yum clean all &>/dev/null && yum makecache &>/dev/null

4.安装数据库–仅192.168.66.10

##下载mysql
	wget http://repo.mysql.com/yum/mysql-5.7-community/el/7/x86_64/mysql57-community-release-el7-10.noarch.rpm

## 添加数据库用户，组mysql
    groupadd mysql 
    useradd -r -g mysql mysql 

## 解压文件，重命名
    tar -xf mysql-5.7.27-el7-x86_64.tar.gz -C /usr/local/ 
    mv /usr/local/mysql-5.7.27-el7-x86_64/ /usr/local/mysql
    chown -R mysql:mysql /usr/local/mysql/
    rm -rf /var/lib/mysql && mkdir -p /var/lib/mysql
    chown mysql:mysql /var/lib/mysql

## 启动配置，开机自启
    rm -rf /etc/init.d/mysql  ##删除之前开机启动的mysql配置
    cp /usr/local/mysql/support-files/mysql.server /etc/init.d/mysql
    echo "basedir=/usr/local/mysql" >>/etc/init.d/mysql
    echo "datadir=/usr/local/mysql/data" >>/etc/init.d/mysql
    chkconfig --add mysql
    chkconfig mysql on

## 数据库配置

```bash
		vim /etc/my.cnf
#——————————————————
[mysqld]
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
port = 3306
socket=/var/lib/mysql/mysql.sock
character-set-server=utf8
 
transaction-isolation = READ-COMMITTED
symbolic-links = 0
server_id=1
max_connections = 5500
log_bin=/var/lib/mysql/mysql_binary_log
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
default_password_lifetime=0
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M 
[client]
default-character-set=utf8
socket=/var/lib/mysql/mysql.sock  
[mysql]
default-character-set=utf8
socket=/var/lib/mysql/mysql.sock
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
#——————————————————————

##环境变量
        echo "export MYSQL_PATH=/usr/local/mysql" >>/etc/profile
        echo "export PATH=\$PATH:\$MYSQL_PATH/bin" >>/etc/profile

数据库初始化

/usr/local/mysql/bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data &>/mysql_pwd.text

## /mysql_pwd.text 可查看初始密码

启动数据库

/etc/init.d/mysql restart

修改密码

mysql_init_pwd="$(grep 'root@localhost' /mysql_pwd.text | awk -F' ' '{print $NF}')"  ##软件初始化的密码  查看替换即可
##CDH6.2_lczy  ##默认设置的密码
/usr/local/mysql/bin/mysql -p${mysql_init_pwd} --connect-expired-password -e "ALTER USER 'root'@'localhost' IDENTIFIED BY 'CDH6.2_lczy';flush privileges"

准备-mysql-connector-java.jar

## 下载一个 mysql-connector-java-5.1.46.jar 修改名字放在服务器目录下/usr/share/java/mysql-connector-java.jar

wget -c -P /usr/share/java "${file_download_url}"/mysql-connector-java.jar &>/dev/null
ls /usr/share/java/mysql-connector-java.jar

创建cdh组件使用的数据库

在这里插入图片描述

CREATE DATABASE amon DEFAULT CHARACTER SET utf8;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8;
##参考
CREATE DATABASE scm DEFAULT CHARACTER SET utf8;
GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%'IDENTIFIED BY 'scm';
FLUSH PRIVILEGES;"

5.安装scm，进入图形化界面

yum install -y cloudera-manager-daemons cloudera-manager-server &>/dev/null

##--下载到本地 /opt/目录下--功能实现上面和上面重复了，建议还是操作一下。
mkdir -p /opt/cloudera/parcel-repo/
wget -c -P /opt/cloudera/parcel-repo "${file_http_url}"/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel  ##"${file_http_url}"当前服务器的ip------http://192.168.66.10/cloudera-repos 
wget -c -P /opt/cloudera/parcel-repo "${file_http_url}"/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1 
wget -c -P /opt/cloudera/parcel-repo "${file_http_url}"/cdh6/6.2.0/parcels/manifest.json 
cp /opt/cloudera/parcel-repo/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha

##添加用户组，权限等
groupadd cloudera-scm 
useradd -r -g cloudera-scm cloudera-scm 
chown cloudera-scm.cloudera-scm /opt/cloudera/parcel-repo/*

##初始化数据库
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh -h localhost mysql scm scm scm | tee /scm_prepare_database.log   ##查看日志/scm_prepare_database.log
#参数说明：
#options 指定操作，如果数据库不再本地请用-h或--host 指定mysql的host，不指定默认localhost
#databaseType 指定为mysql，也可以是其它类型的数据，例如：oracle等
#databaseName 指定为scm数据库，这里使用 scm库
#databaseUser 指定mysql用户名，这里使用 scm
#password 指定mysql其用户名的密码，这里使用scm

##启动
systemctl start cloudera-scm-server
systemctl enable cloudera-scm-server

访问：7180端口
http://192.168.66.10:7180

##在Web浏览器数据 http://192.168.66.10:7180
#登录Cloudera Manager Admin Console，默认凭证为
#Username: admin
#Password: admin

6.其他方式安装：脚本安装一键启动

直接运行该脚本，然后等日志刷新，打开7180进入可视化安装即可。

sh cdh_main.sh "192.168.66.21 192.168.66.22 192.168.66.23" "111111" "22" "http://192.168.38.38:81/cdh"

在这里插入图片描述

脚本下载地址
https://download.csdn.net/download/fzxiaocai/83749872?spm=1001.2014.3001.5503
大家可以相信这份脚本。简单的截图给大家展示一下

7.新增主机

第一步：初始化

sh cdh_init.sh "mysql.cdh.lczy.com" "${file_download_url}"
## 第一个参数：需要设置的主机名
## 第二个参数： 下载安装jdk oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm的下载地址
## 如果本地存在该文件，上传到服务器，注释脚本改行即可。如下图所示
## 可在此链接下载该文件 https://download.csdn.net/download/fzxiaocai/84407632

在这里插入图片描述
第二步：可视化界面扫描主机即可

————————————————————————————end————————————————————————————

沪上求之

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
CDH6.2.0部署

CDH6.2.0部署服务器信息服务器主机名安装服务密码等信息192.168.66.10mysql.cdh.lczy.comcloudera-manager-server httpd mysql-5.7数据库初始密码和安装路径信息如下: mysql_root_pwd=“CDH6.2_lczy” mysql_install_path="/usr/local"192.168.66.11cdh11.lczy.com简单初始化 jdk192.168.66.12cd
复制链接

扫一扫