Centos7 安装Cm6.3.1+Cdh6.3.2

已于 2023-02-01 14:42:52 修改
阅读量1.3k 收藏 3
点赞数
分类专栏： BigData 文章标签： CDH
于 2020-06-18 15:47:08 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/qq_18453581/article/details/106835760
版权

资源合集地址：

安装jdk

rpm -ivh jdk-8u321-linux-x64.rpm
echo -e "# JAVA_HOME \nexport JAVA_HOME=/usr/java/jdk1.8.0_321-amd64 \nexport PATH=$PATH:$JAVA_HOME/bin" >> /etc/profile

安装mysql

wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm
yum -y install mysql57-community-release-el7-10.noarch.rpm 
rpm --import https://repo.mysql.com/RPM-GPG-KEY-mysql-2022
yum -y install mysql-community-server
service mysqld start
systemctl enable mysqld
 
获取mysql临时密码
grep 'temporary password' /var/log/mysqld.log
登录mysql
mysql -uroot -p

修改密码策略
set global validate_password_policy=0; 
set global validate_password_length=1;
ALTER USER 'root'@'localhost' IDENTIFIED BY '123456';
create database scm default character set utf8 default collate utf8_general_ci;
create database hive default character set utf8 default collate utf8_general_ci;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
FLUSH  PRIVILEGES;

使用ntp同步时间，服务端

yum install -y ntpdate ntp
vi /etc/ntp.conf
最下面添加
restrict 192.168.5.0 mask 255.255.255.0 nomodify #运行哪个子网时间同步
server 127.127.1.0
fudge 127.127.1.0 stratun 10
注释掉这些
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

保存退出
systemctl start  ntpd
systemctl enable ntpd

其他客户端

yum install -y ntpdate ntp
vi /etc/ntp.conf
最下面添加
server    master#填写服务器ip或者地址
注释掉这些
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

保存退出
crontab -e
添加
*/2 * * * * sudo ntpdate master
保存退出

先初始化服务器然后修改hosts文件，每个服务器上都要配置

vim /etc/hosts
127.0.0.1   localhost
192.168.5.10    master
192.168.5.11    node1
192.168.5.12    node2

Linux服务器免密登录

ssh-keygen -t rsa
cd /root/.ssh/
ssh-copy-id master
输入 yes然后输入服务器密码
ssh-copy-id node1（同上）
ssh-copy-id node2（同上）
yum -y install httpd createrepo
systemctl start httpd
systemctl enable httpd

 createrepo .命令是将当前目录rpm添加到仓库，可以使用yum install 安装

创建目录

上传资源包到任意目录

mkdir -p /var/www/html/cloudera-repos

unzip -d /var/www/html/cloudera-repos cm6.3.1.zip

chmod 777 -R /var/www/html/cloudera-repos/

createrepo /var/www/html/cloudera-repos/cm6.3.1

创建yum源

cd /etc/yum.repos.d
vim  cloudera-manager.repo

内容如下

[cloudera-manager]
name=Cloudera Manager 6.3.1
baseurl=http://master/cloudera-repos/cm6.3.1
gpgcheck=0
enabled=1

将 /etc/yum.repos.d/cloudera-manager.repo复制到其他节点上

yum clean all && yum makecache
yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

 正常情况,主节点安装,然后使用cm安装,其他节点会自动安装,如果无法自动安装,需要手动在其他从节点安装agent

mkdir -p /opt/cloudera

unzip  -d /opt/cloudera/ parcel-repo.zip

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

把下图需要的文件移进来

 上传到mysql-connector-java.jar /usr/share/java/;mysql驱动名必须改成mysql-connector-java.jar

mkdir -p /usr/share/java/
mv mysql-connector-java.jar /usr/share/java/

创建所有需要的数据库(如果报密码策略问题,需要进入mysql后,执行sql语句创建库表)

mysql -uroot -p123456 -e "set global validate_password_policy=0; 
set global validate_password_length=1;
CREATE DATABASE if not exists scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm';
GRANT ALL ON scm.* TO 'scm'@'localhost' IDENTIFIED BY 'scm';
CREATE DATABASE if not exists  amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon';
GRANT ALL ON amon.* TO 'amon'@'localhost' IDENTIFIED BY 'amon';
CREATE DATABASE if not exists rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman';
GRANT ALL ON amon.* TO 'amon'@'localhost' IDENTIFIED BY 'amon';
CREATE DATABASE if not exists hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; 
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue';
GRANT ALL ON hue.* TO 'hue'@'localhost' IDENTIFIED BY 'hue';
CREATE DATABASE if not exists metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON metastore.* TO 'hive'@'localhost' IDENTIFIED BY 'hive';
CREATE DATABASE if not exists  sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;   
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry';
GRANT ALL ON sentry.* TO 'sentry'@'localhost' IDENTIFIED BY 'sentry';
CREATE DATABASE if not exists nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;      
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav';
GRANT ALL ON nav.* TO 'nav'@'localhost' IDENTIFIED BY 'nav';
CREATE DATABASE if not exists navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms';
GRANT ALL ON navms.* TO 'navms'@'localhost' IDENTIFIED BY 'navms';
CREATE DATABASE if not exists oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
GRANT ALL ON oozie.* TO 'oozie'@'localhost' IDENTIFIED BY 'oozie';
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
FLUSH PRIVILEGES;"

修改hive编码支持中文

#修改字段注释字符集
alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
#修改表注释字符集
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(20000) character set utf8;
#修改分区参数，支持分区建用中文表示
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(20000) character set utf8;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(20000) character set utf8;
#修改索引名注释，支持中文表示
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
#修改视图，支持视图中文
ALTER TABLE TBLS modify COLUMN VIEW_EXPANDED_TEXT mediumtext CHARACTER SET utf8;
ALTER TABLE TBLS modify COLUMN VIEW_ORIGINAL_TEXT mediumtext CHARACTER SET utf8;

mysql数据库与CM Server是同一台主机

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root 123456

(执行报错ssl问题,需关闭mysql ssl)

echo -e "# disable_ssl\nskip_ssl\n" >> /etc/my.cnf
service mysqld restart

mysql数据库与CM Server不在同一台主机上

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root 123456 --scm-host master scm scm scm

启动服务

systemctl start cloudera-scm-server

日志

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

 启动报错解决办法

安装CDH后启动cloudera-scm-server遇到问题_lvtula的博客-CSDN博客

访问http:192.168.5.10/7180  admin/admin

自定义存储库地址

http://master/cloudera-repos/cm6.3.1

检查服务问题

 解决方法每个节点都加

sysctl vm.swappiness=10
cat /proc/sys/vm/swappiness
echo 'vm.swappiness=10'>> /etc/sysctl.conf
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
vi /etc/rc.local
最后一行后面添加
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Root无hdfs权限问题

1、在Linux执行如下命令增加supergroup

groupadd supergroup

2、如将用户root增加到supergroup中

usermod -a -G supergroup root

3、同步系统的权限信息到HDFS文件系统

sudo -u hdfs hdfs dfsadmin -refreshUserToGroupsMappings

4、查看属于supergroup用户组的用户

grep 'supergroup:' /etc/group

hive解决中文乱码

在hive表初始化后,修改表编码

 #修改字段注释字符集
 alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
 #修改表注释字符集
 alter table TABLE_PARAMS modify column PARAM_VALUE varchar(20000) character set utf8;
 #修改分区参数，支持分区建用中文表示
 alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(20000) character set utf8;
 alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(20000) character set utf8;
 #修改索引名注释，支持中文表示
 alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
 #修改视图，支持视图中文
 ALTER TABLE TBLS modify COLUMN VIEW_EXPANDED_TEXT mediumtext CHARACTER SET utf8;
 ALTER TABLE TBLS modify COLUMN VIEW_ORIGINAL_TEXT mediumtext CHARACTER SET utf8;

一键卸载脚本

#!/bin/bash
# 停止CM服务
service cloudera-scm-server stop
service cloudera-scm-agent stop
 
# 卸载CM软件包
yum -y remove cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
 
# 卸载装载点,执行一遍好像不生效
umount cm_processes
umount cm_processes
umount cm_processes
 
# 删除文件
rm -rf /var/log/*
#如果想完全卸载,可以打开注释,只是重装,不需要删除
#rm -rf /opt/cloudera*
rm -rf /etc/systemd/system/multi-user.target.wants/cloudera*
rm -rf /etc/default/cloudera*
rm -rf /etc/cloudera*
rm -rf /var/lib/cloudera*
rm -rf /var/log/cloudera*
rm -rf /usr/lib/systemd/system/cloudera*
rm -rf /run/cloudera*
rm -rf /sys/fs/cgroup/systemd/system.slice/cloudera*
rm -rf /etc/security/limits.d/cloudera*
rm -rf /var/lib/yum/repos/x86_64/7/cloudera*
rm -rf /var/cache/yum/x86_64/7/cloudera*
rm -rf /tmp/*
 
rm -rf /var/lib/hadoop-* 
rm -rf /var/lib/impala 
rm -rf /var/lib/solr 
rm -rf /var/lib/zookeeper 
rm -rf /var/lib/hue 
rm -rf /var/lib/oozie 
rm -rf /var/lib/pgsql 
rm -rf /var/lib/sqoop2 
rm -rf /data/dfs/  
rm -rf /data/impala/ 
rm -rf /data/yarn/  
rm -rf /dfs/ 
rm -rf /impala/ 
rm -rf /yarn/  
rm -rf /var/run/hadoop-*/ 
rm -rf /var/run/hdfs-*/ 
rm -rf /usr/bin/hadoop* 
rm -rf /usr/bin/zookeeper* 
rm -rf /usr/bin/hbase* 
rm -rf /usr/bin/hive* 
rm -rf /usr/bin/hdfs 
rm -rf /usr/bin/mapred 
rm -rf /usr/bin/yarn 
rm -rf /usr/bin/sqoop* 
rm -rf /usr/bin/oozie 
rm -rf /etc/hadoop* 
rm -rf /etc/zookeeper* 
rm -rf /etc/hive* 
rm -rf /etc/hue 
rm -rf /etc/impala 
rm -rf /etc/sqoop* 
rm -rf /etc/oozie 
rm -rf /etc/hbase* 
rm -rf /etc/hcatalog 
 
rm -rf /var/lib/alternatives/impala-conf
rm -rf /var/lib/alternatives/impalad
rm -rf /var/lib/alternatives/impala-collect-diagnostics
rm -rf /var/lib/alternatives/impala-shell
rm -rf /var/lib/alternatives/impala-collect-minidumps
 
rm -rf /etc/alternatives/impala-shell
rm -rf /etc/alternatives/impalad
rm -rf /etc/alternatives/impala-collect-diagnostics
rm -rf /etc/alternatives/impala-conf
rm -rf /etc/alternatives/impala-collect-minidumps
 
rm -rf /var/log/impala*
 
rm -rf /var/lib/alternatives/zookeeper-client
rm -rf /var/lib/alternatives/zookeeper-server
rm -rf /var/lib/alternatives/zookeeper-conf
rm -rf /var/lib/alternatives/zookeeper-server-initialize
rm -rf /var/lib/alternatives/zookeeper-server-cleanup
rm -rf /var/lib/alternatives/zookeeper-security-migration
 
rm -rf /etc/alternatives/zookeeper-conf
rm -rf /etc/alternatives/zookeeper-server
rm -rf /etc/alternatives/zookeeper-server-cleanup
rm -rf /etc/alternatives/zookeeper-server-initialize
rm -rf /etc/alternatives/zookeeper-security-migration
rm -rf /etc/alternatives/zookeeper-client
rm -rf /var/log/zookeeper
 
# 重建数据库
mysql -uroot -p123456 -e "drop database if exists scm;drop database if exists hive;drop database if exists oozie;drop database if exists hue;create database scm DEFAULT CHARACTER SET utf8;create database hive DEFAULT CHARACTER SET utf8;create database oozie DEFAULT CHARACTER SET utf8;create database hue DEFAULT CHARACTER SET utf8;"
 
# 删除用户和组
userdel -r cloudera-scm
userdel -r ambari-qa  
userdel -r appmon  
userdel -r falcon  
userdel -r gpadmin  
userdel -r hcat  
userdel -r hive        
userdel -r kafka  
userdel -r mapred     
userdel -r spark  
userdel -r tez   
userdel -r yarn      
userdel -r zookeeper
userdel -r ams        
userdel -r atlas   
userdel -r flume   
userdel -r hbase    
userdel -r hdfs  
userdel -r infra-solr  
userdel -r livy   
userdel -r mongodb  
userdel -r oozie  
userdel -r sqoop  
userdel -r zeppelin
userdel -r impala
userdel -r kudu
userdel -r hadoop
 
groupdel hadoop
groupdel cloudera-scm
groupdel ambari-qa  
groupdel appmon  
groupdel falcon  
groupdel gpadmin  
groupdel hcat  
groupdel hive        
groupdel kafka  
groupdel mapred     
groupdel spark  
groupdel tez   
groupdel yarn      
groupdel zookeeper
groupdel ams        
groupdel atlas   
groupdel flume   
groupdel hbase    
groupdel hdfs  
groupdel infra-solr  
groupdel livy   
groupdel mongodb  
groupdel oozie  
groupdel sqoop  
groupdel zeppelin
groupdel impala
groupdel kudu
Zakza
关注
专栏目录
基于Centos7搭建CDH6.3.2环境
我的博客
07-07 1343
1、基础环境 OS：Centos 7.6 Cloudera Manager：6.3.1 CDH：6.3.2 JDK：1.8 MySQL（MariaDB）：5.7 设有三个节点 node01 192.168.0.1 node02 192.168.0.2 node03 192.168.0.3 2、服务器设置（所有节点） 2-1、设置静态ip 2-2、修改hostname以及hosts文件 2-3、关闭防火墙 2-4、禁用SELinux 2-4、禁用Swap 2-5、配置免密登录 2-6、安装jdk #安
大数据技术之CM6.3.1+CDH6.3.2 集成 Atlas2.2.0.pdf
11-23
大数据技术之CM6.3.1+CDH6.3.2 集成 Atlas2.2.0.pdf 本文档主要介绍了如何将CM6.3.1和CDH6.3.2集成到Atlas2.2.0中，实现大数据技术的集成和应用。下面是相关知识点的总结： 1. Atlas2.2.0的安装和配置： Atlas是...
centos7安装CM+CDH
qq354436218的博客
01-02 288
http://www.fblinux.com/?p=1492
CentOS 7 部署 CDH 6.3.2
最新发布
a120608yby的博客
08-04 959
CentOS 7 部署 CDH 6.3.2
centos7安装cdh6.3.2-附带安装包
catcher92的博客
09-15 611
基于centos7安装cdh6.3.2，附带百度云盘安装包，100%可成功
在centos7上 安装CM的步骤
JCY19890312的博客
08-30 842
1. 安装jdk    （http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_jdk_installation.html） 2. 安装mysql  （http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_mysql.h
centos7环境下安装CM、CDH搭建大数据集群(未完)
phenixzhong的专栏
10-21 576
、浏览器地址栏中输入：localhost:7180(因为VMware中浏览器不能显示所有网页，所以在宿主机的浏览器中输入192.168.40.11:7180)；输入用户名、密码(都是admin) 、登录进去后的界面如下： 、点击上图中的Continue按钮后的界面如下： 、修改上图中的集群名和点击Continue按钮后的界面如下： 、在上图中输入你要管理的机器名(/etc/hosts中要保存有机器名和ip地址的映射)后，点击search按钮，结果见下图： 、点击上图中的C...
大数据技术之CM6.3.1+CDH6.3.2配置Hue+Sentry权限管理.pdf
11-23
大数据技术之CM6.3.1+CDH6.3.2配置Hue+Sentry权限管理 本文档主要介绍了CDH6.3.2配置Hue+Sentry权限管理的步骤，旨在帮助读者了解大数据技术中的权限管理机制。 一、Sentry概述 Sentry是Apache开源组件，提供了细...
大数据技术之CM6.3.1+CDH6.3.2集群模式部署Dolphinscheduler.pdf
11-23
大数据技术之CM6.3.1+CDH6.3.2集群模式部署Dolphinscheduler 大数据技术之CM6.3.1+CDH6.3.2集群模式部署Dolphinscheduler是大数据领域的一种集群模式部署解决方案。该解决方案使用CDH6.3.2集群模式，结合Dolphin...
大数据技术之Centos7.6部署CM6.3.1+CDH6.3.2集群.pdf
11-23
Centos7.6部署CM6.3.1+CDH6.3.2集群.pdf
CM6.3.1和CDH6.3.2资源下载
04-14
本资源包含redhat7版本的CM6.3.1和CDH6.3.2： manifest.json cm6.3.1-redhat7.tar.gz CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256 allkeys.asc
CDH6.3.2+centos7完全离线下安装手册
09-12
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256 manifest.json CentOS-7-x86_64-DVD-1611.iso cm6.3.1-redhat7.tar.gz jdk-8u251-linux-x64.tar.gz mysql-5.7.25-1.el7.x86_64.rpm-bundle.tar mysql-connector-java-5.1.46.tar.gz 离线安装CDH6.3.2.docx
Cloudera CM6.3.1 CDH6.3.2 完整版+离线安装教程.txt
05-11
Cloudera CM6.3.1 CDH6.3.2 完整版+离线安装教程
cdh6.3.1+cm6.3.1安装包-cdh安装包大全
09-05
cdh6.3.1 + cm6.3.1版本网盘地址，操作系统为Redhat7和Centos7及以上，链接失效可私信，其他版本也可以私信
CDH6.3安装指南.pdf
09-11
最详细的CDH6.3安装指南，可以一步步按照文档进行安装，并且整理了安装介质（通过百度网盘下载）。
小白从CentOS7到安装CDH6.3.2入坑实操指北(二)
quantam的博客
09-20 836
CDH集群安装和问题处理过程
CDH 6.3.2-1 离线安装部署（Centos7.9）
小强签名设计 的博客
12-27 1073
  三台服务器都操作： 三、安装 Cloudera-Manager 四、访问 Cloudera Manager 五、集群异常信息解决 参考： Linux搭建Cloudera Manager+CDH6.3.1环境
CDH6.3.2安装包及安装教程网盘下载地址.txt
04-10
从2021年1月31日开始，所有Cloudera软件都需要有效的订阅，并且只能通过付费墙进行访问。也就是说CDH各版本都不能免费获取了。 安装教程中详细说明了安装过程中遇到的各种问题，并且整个过程不需要依赖外网，全部为离线安装。 网盘下载《CDH6.3.2安装包及安装教程网盘下载地址.txt》，包括： manifest.json CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel.sha CentOS7.6下CDH6.3.2离线安装教程(正式环境).docx
评论 2
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值