cloudera-manager安装+CDH5.4.1部署+mysql+python+impala安装

最新推荐文章于 2023-11-09 11:31:55 发布

老农民挖数据

最新推荐文章于 2023-11-09 11:31:55 发布

阅读量4.6k

点赞数

分类专栏： hadoop 文章标签： hadoop集群搭建

本文链接：https://blog.csdn.net/shushugood/article/details/50628125

版权

hadoop 专栏收录该内容

37 篇文章 0 订阅

订阅专栏

三台Red Hat Enterprise Linux Server release 6.3 (Santiago) ，搭建hadoop

[root@master01 cdh_install]# more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.5.140 master01
192.168.5.141 slave01
192.168.5.142 slave02

--解压jdk

tar -xvf jdk-7u7-linux-i586.tar.gz

--配置jdk环境变量

export JAVA_HOME=/it/jdk1.7.0_07
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH

--测试java环境

[root@master01 it]# source /etc/profile
[root@master01 it]# java -version
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) Server VM (build 23.3-b01, mixed mode)

--上传安装包

[root@master01 cdh_install]# mkdir -p cdh_install
[root@master01 cdh_install]# cd cdh_install/

[root@master01 cdh_install]# scp * root@slave01:/it/cdh_install/
root@slave01's password: 
CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel                                  100% 1215MB  46.7MB/s   00:26    
CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha                              100%   41     0.0KB/s   00:00    
cloudera-manager-el6-cm5.4.1_x86_64.tar.gz                            100%  659MB  43.9MB/s   00:15    
manifest.json                                                         100%   41KB  41.5KB/s   00:00    
Python-2.6.6.tgz                                                      100%   13MB  12.7MB/s   00:00    
Python-2.7.10.tgz                                                     100%   16MB  16.0MB/s   00:01    
[root@master01 cdh_install]# scp * root@slave02:/it/cdh_install/
root@slave02's password: 
CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel                                  100% 1215MB  40.5MB/s   00:30    
CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha                              100%   41     0.0KB/s   00:00    
cloudera-manager-el6-cm5.4.1_x86_64.tar.gz                            100%  659MB  47.1MB/s   00:14    
manifest.json                                                         100%   41KB  41.5KB/s   00:00    
Python-2.6.6.tgz                                                      100%   13MB  12.7MB/s   00:00    
Python-2.7.10.tgz                                                     100%   16MB  16.0MB/s   00:01    
[root@master01 cdh_install]#

--下面我们进行CM5.4.1的安装配置，在安装的过程中，官方文档有点累赘，看起来有点费劲（但不可否认，它确实很详细，最权威的文档）

--Installation Path C - Manual Installation Using Cloudera Manager Tarballs

--建立目录，拷贝文件(root用户下执行以下命令)
sudo mkdir /opt/cloudera-manager
tar xzf cloudera-manager*.tar.gz -C /opt/cloudera-manager
useradd --system --home=/opt/cloudera-manager/cm-5.4.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm


mkdir -p /var/log/cloudera-scm-headlamp
mkdir -p /var/log/cloudera-scm-firehose
mkdir -p /var/log/cloudera-scm-alertpublisher
mkdir -p /var/log/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-headlamp
mkdir -p /var/lib/cloudera-scm-firehose
mkdir -p /var/lib/cloudera-scm-alertpublisher
mkdir -p /var/lib/cloudera-scm-eventserver
mkdir -p /var/lib/cloudera-scm-server


chown -R cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp
mkdir -p /var/cm_logs/cloudera-scm-headlamp
chown cloudera-scm /var/cm_logs/cloudera-scm-headlamp
mkdir -p /opt/cloudera/parcel-repo
chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels

--下面进行mysql的安装配置（rpm包）

--mysql安装

[root@master01 cdh_install]#  rpm -ivh mysql-community-client-5.7.9-1.el6.x86_64.rpm mysql-community-com-embedded-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-devel-5.7.9-1.el6.x86_64.rpm mysql-community-lmunity-server-5.7.9-1.el6.x86_64.rpm
warning: mysql-community-client-5.7.9-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: N
Preparing...                ########################################### [100%]
	file /usr/share/mysql/czech/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_64
	file /usr/share/mysql/danish/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_6
	file /usr/share/mysql/dutch/errmsg.sys from install of mysql-community-common-5.7.9-1.el6.x86_64

上面mysql安装报错，肯定是mysql有低版本的包，查看mysql已安装的包

[root@master01 cdh_install]# rpm -qa | grep -i mysql
mysql-libs-5.1.61-4.el6.x86_64

--卸载已安装的mysql

[root@master01 cdh_install]# yum -y remove mysql-libs-5.1.61*
Loaded plugins: product-id, refresh-packagekit, subscription-manager
Updating certificate-based repositories.
Unable to read consumer identity
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package mysql-libs.x86_64 0:5.1.61-4.el6 will be erased
--> Processing Dependency: libmysqlclient.so.16()(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64
--> Processing Dependency: libmysqlclient.so.16(libmysqlclient_16)(64bit) for package: 2:postfix-2.6.6-2.2.el6_1.x86_64
--> Processing Dependency: mysql-libs for package: 2:postfix-2.6.6-2.2.el6_1.x86_64
--> Running transaction check
---> Package postfix.x86_64 2:2.6.6-2.2.el6_1 will be erased
--> Processing Dependency: /usr/sbin/sendmail for package: cronie-1.4.4-7.el6.x86_64
--> Running transaction check

省略。。。

Removed:
  mysql-libs.x86_64 0:5.1.61-4.el6                                                                      

Dependency Removed:
  cronie.x86_64 0:1.4.4-7.el6       cronie-anacron.x86_64 0:1.4.4-7.el6  crontabs.noarch 0:1.10-33.el6 
  postfix.x86_64 2:2.6.6-2.2.el6_1  sysstat.x86_64 0:9.0.4-20.el6       

Complete!
[root@master01 cdh_install]#

--查看是否已卸载

[root@master01 cdh_install]# rpm -qa | grep -i mysql
[root@master01 cdh_install]# 
没有了

--重新安装

[root@master01 cdh_install]#  rpm -ivh mysql-community-client-5.7.9-1.el6.x86_64.rpm mysql-community-common-5.7.9-1.el6.x86_64.rpm mysql-community-devel-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-5.7.9-1.el6.x86_64.rpm mysql-community-embedded-devel-5.7.9-1.el6.x86_64.rpm mysql-community-libs-5.7.9-1.el6.x86_64.rpm mysql-community-libs-compat-5.7.9-1.el6.x86_64.rpm mysql-community-server-5.7.9-1.el6.x86_64.rpm 
warning: mysql-community-client-5.7.9-1.el6.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                ########################################### [100%]
   1:mysql-community-common ########################################### [ 13%]
   2:mysql-community-libs   ########################################### [ 25%]
   3:mysql-community-client ########################################### [ 38%]
   4:mysql-community-devel  ########################################### [ 50%]
   5:mysql-community-embedde########################################### [ 63%]
   6:mysql-community-embedde########################################### [ 75%]
   7:mysql-community-server ########################################### [ 88%]
   8:mysql-community-libs-co########################################### [100%]
[root@master01 cdh_install]#

--重设置密码

5.7.9重设置密码
有人会问，为什么user没有password字段了。从5.6之后就没有了


会在  /var/log/mysqld.log  最后生成一个随机密码，供初始登录。
[root@master01 mysql]# more /var/log/mysqld.log 
160203 14:40:45 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2016-02-03T06:40:45.397816Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use 
--explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-03T06:40:45.399265Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.9) starting as process 36627 ...
2016-02-03T06:40:45.407902Z 0 [Note] InnoDB: PUNCH HOLE support available
2016-02-03T07:28:24.812608Z 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.7.9'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2016-02-03T07:50:54.238157Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2016-02-03T07:50:54.708935Z 0 [Warning] InnoDB: New log files created, LSN=45790
2016-02-03T07:50:54.780046Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2016-02-03T07:50:54.840130Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: da97342a-ca4a-11e5-
acfb-000c29c14e9c.
2016-02-03T07:50:54.841328Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2016-02-03T07:50:54.845873Z 1 [Note] A temporary password is generated for root@localhost: C*s!.=y6<-OC
[root@master01 mysql]#    mysql -uroot -p
password:

--或者直接跳过权限控制，修改密码（我通常采用这种）

[root@master01 mysql]# mysqld_safe --skip-grant-tables --user=mysql & 
[1] 37043
[root@master01 mysql]# 160203 15:59:11 mysqld_safe Logging to '/var/log/mysqld.log'.
160203 15:59:11 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.9 MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

mysql> 

mysql> use mysql
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.
mysql> SET PASSWORD = PASSWORD('123456'); 
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql>  FLUSH PRIVILEGES; 
Query OK, 0 rows affected (0.00 sec)

mysql> 
mysql> 
mysql> exit
Bye

--我们登陆试试

[root@master01 mysql]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.7.9 MySQL Community Server (GPL)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit

搞定。

--重启mysqld

[root@master01 mysql]# vi /etc/my.cnf  --调整一些参数
[root@master01 mysql]# sudo /sbin/chkconfig mysqld on
[root@master01 mysql]# sudo /sbin/chkconfig --list mysqld
mysqld         	0:off	1:off	2:on	3:on	4:on	5:on	6:off

[root@master01 mysql]# sudo service mysqld start
Starting mysqld:                                           [  OK  ]

--把jar包拷贝到对应目录下，改为为mysql-connector-java.jar

[root@master01 share]# mkdir -p /usr/share/java
[root@master01 share]# sudo cp /it/cdh_install/mysql-connector-java-5.1.38-bin.jar  /usr/share/java/mysql-connector-java.jar
[root@master01 share]# cd /usr/share/java
[root@master01 java]# ls
mysql-connector-java.jar 
[root@master01 share]#

--建立所需用户

mysql> create database amon DEFAULT CHARACTER SET utf8;
Query OK, 1 row affected (0.00 sec)
mysql> grant all PRIVILEGES on amon.* TO ' amon'@'%' IDENTIFIED BY ' 123456' WITH GRANT OPTION;
Query OK, 1 row affected (0.00 sec)
 
省略（
省略create database ....
GRANT ALL PRIVILEGES ON scm.* TO 'scm'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
GRANT ALL PRIVILEGES ON rman.* TO 'rman'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
grant all PRIVILEGES on metastore.* TO 'hive'@'%' IDENTIFIED BY '123456' WITH GRANT OPTION;
grant all PRIVILEGES on sentry.* TO 'sentry'@'%' IDENTIFIED BY '123456'  WITH GRANT OPTION;
grant all PRIVILEGES on nav.* TO 'nav'@'%' IDENTIFIED BY '123456'  WITH GRANT OPTION;
grant all PRIVILEGES on navms.* TO 'navms'@'%' IDENTIFIED BY '123456'  WITH GRANT OPTION;

）

mysql> 
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

--备份mysql

[root@master01 java]# mysqldump  -uroot -p mysql > /tmp/mysql-bakup.sql
Enter password: 
[root@master01 java]# cd /tmp/
[root@master01 tmp]# ls
hsperfdata_root  mysql-bakup.sql  pulse-ahmL5Yupk46M  ssh-XKnrmy7U7AfM
keyring-NkHAG2   orbit-root       pulse-Nij3doz66iv1  yum.log

----建立scm

[root@master01 schema]# pwd
/opt/cloudera-manager/cm-5.4.1/share/cmf/schema
[root@master01 schema]# ./scm_prepare_database.sh mysql scm scm 123456
JAVA_HOME=/it/jdk1.7.0_07
Verifying that we can write to /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-server
Executing:  /it/jdk1.7.0_07/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera-manager/cm-5.4.1/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cloudera-manager/cm-5.4.1/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Wed Feb 03 17:58:37 CST 2016 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!
[root@master01 schema]#

----把对应的文件拷贝到repo下，并启动cm-server

[root@master01 parcel-repo]# pwd
/opt/cloudera/parcel-repo
[root@master01 parcel-repo]# cp /it/cdh_install/CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel* manifest.json .
[root@master01 parcel-repo]# ls -l
total 1244320
-rw-r--r-- 1 root root 1274129311 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel
-rw-r--r-- 1 root root         41 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha
-rw-r--r-- 1 root root      42445 Feb  3 18:04 manifest.json
[root@master01 parcel-repo]# ls -l
total 1244276
-rw-r--r-- 1 root root 1274129311 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel
-rw-r--r-- 1 root root         41 Feb  3 17:16 CDH-5.4.1-1.cdh5.4.1.p0.6-el6.parcel.sha
[root@master01 parcel-repo]# cd /opt/cloudera-manager/cm-5.4.1/etc/init.d/
[root@master01 init.d]# ls
cloudera-scm-agent  cloudera-scm-server  cloudera-scm-server-db
[root@master01 init.d]# ls -l
total 20
-rwxr-xr-x 1 1106 4001 5241 May  9  2015 cloudera-scm-agent
-rwxr-xr-x 1 1106 4001 6105 May  9  2015 cloudera-scm-server
-rwxr-xr-x 1 1106 4001 2894 May  9  2015 cloudera-scm-server-db
[root@master01 init.d]# ./cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]
[root@master01 init.d]#

--查看启动日志

[root@master01 cloudera-scm-server]# tailf cloudera-scm-server.log 
2016-02-03 18:02:51,753 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2016-02-03T10:02:51.753Z
2016-02-03 18:02:51,790 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2016-02-03T10:02:51.790Z
2016-02-03 18:02:51,801 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:114
2016-02-03 18:02:51,801 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2016-02-03T10:02:51.801Z
2016-02-03 18:02:51,838 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:126
2016-02-03 18:02:51,840 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2016-02-03T10:02:51.839Z
2016-02-03 18:02:52,118 INFO WebServerImpl:org.mortbay.log: jetty-6.1.26.cloudera.4
2016-02-03 18:02:52,120 INFO WebServerImpl:org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7180
2016-02-03 18:02:52,120 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.



CM-server就搞定了

-- 启动cm-agent

先检查包 
[root@master01 init.d]# rpm -q  chkconfig  python   bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
chkconfig-1.3.49.3-2.el6.x86_64
python-2.6.6-29.el6_2.2.x86_64
package bind-utils is not installed
psmisc-22.6-15.el6_0.1.x86_64
libxslt-1.1.26-2.el6.x86_64
zlib-1.2.3-27.el6.x86_64
sqlite-3.6.20-1.el6.x86_64
package cyrus-sasl-plain is not installed
package cyrus-sasl-gssapi is not installed
package fuse is not installed
package portmap is not installed
package fuse-libs is not installed
package redhat-lsb is not installed
[root@master01 init.d]#

--yum安装所需要的包

[root@master01 init.d]# yum install chkconfig  python   bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb
Loaded plugins: product-id, refresh-packagekit, subscription-manager

--启动代理

[root@master01 init.d]# ./cloudera-scm-agent start
Starting cloudera-scm-agent:                               [  OK  ]
[root@master01 init.d]#

--可以登录界面，默认用户密码为admin/admin

这里有140，141，142三台机器，我们这里先安装140=master01 141=slave01，

142等安装好后，作为新节点添加。（测试添加节点）

老农民挖数据

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
复制链接

分享到 QQ

分享到新浪微博

扫一扫

专栏目录