hadoop 环境搭建三

最新推荐文章于 2021-01-19 12:08:37 发布

大漠孤烟BLOG

最新推荐文章于 2021-01-19 12:08:37 发布

阅读量400

点赞数

分类专栏： hadoop 环境搭建

本文链接：https://blog.csdn.net/u012842247/article/details/103191087

版权

hadoop 环境搭建专栏收录该内容

4 篇文章 0 订阅

订阅专栏

一、搭建jdk ，这里使用是jdk1.8

//创建java jdk目录
[root@hadoop01 /]# mkdir /usr/java
[root@hadoop01 cdh]# tar -zxvf jdk-8u202-linux-x64.tar.gz  -C /usr/java

[root@hadoop01 java]# chown -R root:root /usr/java/jdk1.8.0_202/
[root@hadoop01 java]# ll
总用量 0
drwxr-xr-x 7 root root 245 12月 16 2018 jdk1.8.0_202
[root@hadoop01 java]# vi /etc/profile
[root@hadoop01 java]# source /etc/profile
//验证是否OK
[root@hadoop01 java]# which java

hadoop02、hadoop03 同理设置

二、在hadoop01 上搭建 mysql。

此处省略，我使用安装好的机器上。

创建cdh 元数据存储空间

create database cmf DEFAULT CHARACTER SET utf8;
create database amon DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'cmf1234!';
grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon1234!';
flush privileges;

三、开始安装cdh设置

1.1.所有节点创建目录及解压
mkdir /opt/cloudera-manager
tar -zxvf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager/

1.2.所有节点修改agent的配置，指向server的节点hadoop01
sed -i "s/server_host=localhost/server_host=hadoop01/g" /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

1.3.主节点修改server的配置:
vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-server/db.properties 
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#

# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
#com.cloudera.cmf.db.host=hadoop03.fandong.com

# The database name
#com.cloudera.cmf.db.name=cmf

# The database user
#com.cloudera.cmf.db.user=cmf

# The database user's password
#com.cloudera.cmf.db.password=cmf1234!

# The db setup type
# By default, it is set to INIT
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
com.cloudera.cmf.db.setupType=EXTERNAL
~                                        

1.4.所有节点创建用户
useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

1.5.目录修改用户及用户组
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

---注解以上参考若泽数据

离线源部署

$ mkdir -p /opt/cloudera/parcel-repo
$ ll
total 3081664
-rw-r--r-- 1 root root 2127506677 May  9 18:04 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root         41 May  9 18:03 CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root  841524318 May  9 18:03 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root  185515842 Aug 10  2017 jdk-8u144-linux-x64.tar.gz
-rw-r--r-- 1 root root      66538 May  9 18:03 manifest.json
-rw-r--r-- 1 root root     989495 May 25  2017 mysql-connector-java.jar
$ cp CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel /opt/cloudera/parcel-repo/

#切记cp时，重命名去掉1，不然在部署过程CM认为如上文件下载未完整，会持续下载
$ cp CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
$ cp manifest.json /opt/cloudera/parcel-repo/

2.2.目录修改用户及用户组
$ chown -R cloudera-scm:cloudera-scm /opt/cloudera/

---启动

//启动server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start

//启动agent
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

//验证端口
http://hadoop01:7180   admin/admin

查看log，发现

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at sun.reflect.GeneratedConstructorAccessor7.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:342)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2197)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2230)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2025)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:778)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:386)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330)
        at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
        at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
        at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
        at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
        at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
        at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
        at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Caused by: java.net.UnknownHostException: null
        at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
        at java.net.InetAddress.getAllByName(InetAddress.java:1193)
        at java.net.InetAddress.getAllByName(InetAddress.java:1127)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:188)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:301)
        ... 19 more
2019-11-21 23:35:28,645 INFO main:org.hibernate.cache.spi.UpdateTimestampsCache: HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache
2019-11-21 23:35:28,730 WARN main:org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.
2019-11-21 23:35:28,741 INFO main:org.hibernate.cache.internal.StandardQueryCache: HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache
2019-11-21 23:35:28,743 WARN main:org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.
2019-11-21 23:35:29,782 INFO main:org.hibernate.validator.internal.util.Version: HV000001: Hibernate Validator 5.0.1.Final
2019-11-21 23:37:01,693 INFO main:com.cloudera.enterprise.CommonMain: Statistics not enabled, JMX will not be registered

经过排查发现，是db.properties 配置文中的属性被注释掉了。去掉# 即可。

随后可以启动，进行界面安装。