CDH-6.0.1 配置&错误解决

最新推荐文章于 2023-08-26 01:33:02 发布

疯飙的蜗牛

最新推荐文章于 2023-08-26 01:33:02 发布

阅读量3.4k

点赞数 3

分类专栏： hadoop 文章标签：大数据

本文链接：https://blog.csdn.net/zhangxueleishamo/article/details/118932999

版权

hadoop 专栏收录该内容

4 篇文章 0 订阅

订阅专栏

一、配置前准备&系统配置

1、服务器准备
        10.2.33.96 cdh.manager1.ops
        10.2.33.97 cdh.data1.ops
        10.2.33.98 cdh.data2.ops
        10.2.33.99 cdh.data3.ops

2、安装包准备

链接：https://pan.baidu.com/s/1Fwma2kbaXLuvYv7J19qrTg
提取码：vtfd

jdk:
oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm

mysql:
   mysql-community-client-5.6.28-2.el7.x86_64.rpm
   mysql-community-devel-5.6.28-2.el7.x86_64.rpm
   mysql-community-release-el7-5.noarch.rpm
   mysql-community-common-5.6.28-2.el7.x86_64.rpm
   mysql-community-libs-5.6.28-2.el7.x86_64.rpm
   mysql-community-server-5.6.28-2.el7.x86_64.rpm
   mysql-connector-java-5.1.47.tar.gz
   mysql-connector-java.jar

cloudera-repos-6.0.1:
   cloudera-manager-daemons-6.0.1-610811.el7.x86_64.rpm
   cloudera-manager-agent-6.0.1-610811.el7.x86_64.rpm
   cloudera-manager-server-db-2-6.0.1-610811.el7.x86_64.rpm
   cloudera-manager-server-6.0.1-610811.el7.x86_64.rpm

parcel-6.0.1
   CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
   CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
   CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.torrent
   manifest.json

3、系统环境配置(所有服务器)
   hostname配置： hostnamectl set-hostname 名字
   免密配置： ssh-keygen; ssh-copy-id -i /root/.ssh/id_rsa.pub root@ip
   关闭防火墙：systemctl stop(disable) firewalld
   关闭selinux: /usr/sbin/setenforce 0; /bin/sed -ri '/^SELINUX=\w+$/ s//SELINUX=disabled/' /etc/selinux/config;
   配置时间服务器: /usr/bin/yum -y install ntp；ntpq -p
   大页存储： echo never > /sys/kernel/mm/transparent_hugepage/defrag ; echo never > /sys/kernel/mm/transparent_hugepage/enabled
   文件句柄：sed -i '/# End of file/ i\* soft nofile 102400' /etc/security/limits.conf
       sed -i '/# End of file/ i\* hard nofile   102400' /etc/security/limits.conf
       sed -i 's/* soft nproc 1024/* soft nproc 65535/g' /etc/security/limits.d/20-nproc.conf

java配置(必须相匹配，如有其它版本需要卸载)：

          yum -y install oracle-j2sdk1.8.x86_64
          vi /etc/profile
            export JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
            export PATH=$JAVA_HOME/bin:$PATH
          source /etc/profile

二、安装配置阶段
注：已把相关rpm上传到了自建的yum库上。(yum库配置看前文https://blog.csdn.net/zhangxueleishamo/article/details/82182676?spm=1001.2014.3001.5502)

1、cloudera安装
manager1上配置
yum -y install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
将parcel-6.0.1 下的4个包放到manager服务器的/opt/cloudera/parcel-repo/路径下(权限很重要)

drwxr-xr-x  2 cloudera-scm cloudera-scm  218 7月  29 14:19 parcel-repo
-rw-r----- 1 cloudera-scm cloudera-scm 1988378222 7月  29 14:19 CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel
-rw-r--r-- 1 cloudera-scm cloudera-scm         41 7月  29 14:19 CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.sha
-rw-r----- 1 cloudera-scm cloudera-scm      76029 7月  29 14:19 CDH-6.0.1-1.cdh6.0.1.p0.590678-el7.parcel.torrent
-rw-r--r-- 1 cloudera-scm cloudera-scm      28173 7月  29 14:19 manifest.json

其它机器上配置
yum -y install cloudera-manager-daemons cloudera-manager-agent

2、数据库配置(manager1)

####数据库连接java文件需放到指定地方，所有服务器都需要；文件在网盘的mysql目录下
mkdir /usr/share/java -p
mv mysql-connector-java.jar /usr/share/java/

####在manager上安装mysql
yum -y install mysql-server

####修改配置文件vim /etc/my.cnf
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
log_bin=/var/lib/mysql/mysql_binary_log
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

####启动服务
systemctl start mysql
systemctl enable mysql

3、创建CDH所需要的数据库
根据所需要安装的服务参照下表创建对应的数据库以及数据库用户，数据库必须使用utf8编码，创建数据库时要记录好用户名及对应密码：

Databases for Cloudera Software
Service	Database	User	key
Cloudera Manager Server	scm	scm	123456
hive	hive	hive	123456
Activity Monitor	amon	amon	123456
Reports Manager	rman	rman	123456
Hue	hue	hue	123456
Hive Metastore Server	metastore	hive	123456
Sentry Server	sentry	sentry	123456
Cloudera Navigator Audit Server	nav	nav	123456
Cloudera Navigator Metadata Server	navms	navms	123456
Oozie	oozie	oozie	123456

#######创建数据库
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
#######赋权
GRANT ALL ON hive.* TO 'hive'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON scm.* TO 'scm'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON amon.* TO 'amon'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON rman.* TO 'rman'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON hue.* TO 'hue'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON metastore.* TO 'metastore'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON sentry.* TO 'sentry'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON nav.* TO 'nav'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON navms.* TO 'navms'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON oozie.* TO 'oozie'@'localhost' IDENTIFIED BY '123456' WITH GRANT OPTION;

GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;

GRANT ALL ON hive.* TO 'hive'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON scm.* TO 'scm'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON amon.* TO 'amon'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON rman.* TO 'rman'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON hue.* TO 'hue'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON metastore.* TO 'metastore'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON sentry.* TO 'sentry'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON nav.* TO 'nav'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON navms.* TO 'navms'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL ON oozie.* TO 'oozie'@'cdh.manager1.ops' IDENTIFIED BY '123456' WITH GRANT OPTION;

####注：‘%’ 没生效，故按照ip&域名赋权
FLUSH PRIVILEGES;

#####cm初始化mysql连接
 /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm 123456

4、修改&查看配置文件
只有manager需要:
需要修改，如初始化连接出问题修改

vim /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=123456

无需修改，但是需要知道有这么个文件
/etc/default/cloudera-scm-server

所有节点都需要：
修改： vim /etc/cloudera-scm-agent/config.ini
server_host=cdh.manager1.ops

5、服务启动,查看日志
注：需注意启动顺序，先开启service,再开启agent, 否则会出现错误; 命令如下，需查看日志中出现的错误，并解决，错误在后面有总结

systemctl start cloudera-scm-service 
systemctl enable cloudera-scm-service 
systemctl start cloudera-scm-agent
systemctl enable cloudera-scm-agent

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

6、Parcel配置，访问server的7180端口http://cdh.manager1.ops:7180 用户名和密码都是admin

该步骤中用户可自定义，但是必须sudo权限，后面继续即可

7、主机安装完成：会对主机进行检查，有个 swappness 需要调整，如果是真实服务器，请将 swappness修改为0，尽量不使用交换分区；添加服务

保持默认的节点角色配置：
选择自定义数据库，填写连接信息，并点击测试连接按钮：

8、进入到cloudera-manager管理页面

三、错误解决
1、报错：Inspector did not run successfully
解决：
ps -ef|grep agent
root 850 1 0 12:17 ? 00:00:11 /usr/bin/python2 /opt/cloudera/cm-agent/bin/../bin/supervisord -n
可能是多次重启server或agent发生错误；也可能是启动服务的时候，找的路径不正确；
将进程杀死，它会自动启动；然后重启agent服务 systemctl restart cloudera-scm-agent
2、报错：IOException thrown while collecting data from host:没有到主机的路由
检查防火墙，可能是主机的防火墙忘记关闭或者禁用，关闭防火墙，关闭selinux; 解决==
3、该主机已与 Cloudera Manager Server 未建立联系
按顺序将组件重启即可
4、错误：ScmActive: Unable to retrieve non-local non-loopback IP address. Seeing address: localhost/127.0.0.1
检查各服务器hosts是否配置正确，检查第二、4步骤中的所有节点server_host是否更改，防火墙是否关闭，检测ip和hostname免密登陆
5、java.sql.SQLException: Access denied for user ‘scm’@‘localhost’ (using password: YES)
解决：注'%'权限中不包含localhost，需再赋权一遍localhost
6、**ERROR Unable to create/drop a table.java.sql.SQLException: Statement violates GTID consistency: CREATE TABLE … SELECT.
解决：连接成功，但是不能建表
若my.cnf中存在gtid_mode、enforce_gtid_consistency字段，需修改为如下内容：
gtid_mode = OFF
enforce_gtid_consistency = OFF
要两个参数一起改,然后重启mysql，问题解决
7、未在已配置的存储库中找到任何 parcel。尝试在更多选项下添加一个自定义存储库。否则，您可能只能继续使用包。
注意： Parcle默认存储目录 /opt/cloudera/parcels
本地存放目录：默认/opt/cloudera/parcel-repo
查看manager本地存放目录中是否包含4个文件，第二、1步骤
重启server和agent，启动顺序需注意，先server后agent
配置修改完最好重启下agent，如还是报错重启服务器
8、agent日志中错误 Error, CM server guid updated, expected XXXX, received XXXX
MainThread parcel INFO Agent does create users/groups and apply file permissions
MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
MainThread agent ERROR Error, CM server guid updated, expected 7cda865f-a833-40d1-9fa6-761631c74326, received e2678b78-7c5e-4703-91b6-c52e2875988c
产生的原因是服务器以前作为集群节点安装过agent服务，再次使用时没有卸载干净
解决办法：
rm -rf /var/lib/cloudera-scm-agent/cm_guid
service cloudera-scm-agent restart
9、其它注意事项：
hostname中指定的集群名，不能有下划线！！！！
防火墙请关闭，这样你可少很多烦恼
执行systemctl start cloudera-scm-agent 和 systemctl start cloudera-scm-server启动失败请使用：journalctl -xe指令仔细查看错误信息。然后百度解决！
jdk别用自己的，要用官网提供的。
保证集群机器的内存足够！！！
第一次安装完成进入，可能很多都没启动成功爆红。根据提示错误信息，修正配置，重启即可！！

10、2021-07-29 14:32:31,360 WARN main:org.springframework.context.support.GenericApplicationContext: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version.
2021-07-29 14:32:31,361 ERROR main:com.cloudera.server.cmf.Main: Server failed.
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Unable to obtain CM release version.

解决：该问题是因为 mysql-connector-java.jar 版本不适合链接数据库，注意改文件权限为755

疯飙的蜗牛

关注

3
点赞
踩
5

收藏

觉得还不错? 一键收藏
打赏
0
评论
CDH-6.0.1 配置&错误解决

一、配置前准备&系统配置1、服务器准备10.2.33.96 cdh.manager1.ops10.2.33.97 cdh.data1.ops10.2.33.98 cdh.data2.ops10.2.33.99 cdh.data3.ops2、安装包准备jdk: oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpmmysql: mysql-community-clie...
复制链接

扫一扫