参考博客
Cloudera Manager安装之利用parcels方式安装单节点集群
Cloudera Manager安装之Cloudera Manager 5.3.X安装(三)(tar方式、rpm方式和yum方式)
ClouderManager官网的安装历史稳定版
https://www.cloudera.com/documentation/manager/5-1-x/Cloudera-Manager-Installation-Guide/cm5ig_install_older_cm5.html
说在前面的话(看清楚就好!!!)
我这篇博客,是两种方式都使用过:(在/etc/yum.repos.d下的)
[hadoop@clouderamanager01 yum.repos.d]$ sudo vi cloudera-manager.repo [hadoop@clouderamanager02 yum.repos.d]$ sudo vi cloudera-manager.repo [hadoop@clouderamanager03 yum.repos.d]$ sudo vi cloudera-manager.repo
即(博主我推荐用这种!!!)(但是不需)
[cloudera-manager] # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 name=Cloudera Manager baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/ gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera gpgcheck = 1
或者(但是,需要自定义存储库)
[cloudera-manager] # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 name=Cloudera Manager baseurl=https://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
enabled = 1 gpgcheck = 1
我这里是,使用CM 5.3.x利用parcels方式集群安装CDH 5.3.x的3节点集群!(开始开工)
http://archive.cloudera.com/cdh5/parcels/5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel http://archive.cloudera.com/cdh5/parcels/5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 http://archive.cloudera.com/cdh5/parcels/5.3.6/manifest.json
以下是clouderamanager01这台机器上(主节点)
[hadoop@clouderaManager01 softwares]$ pwd /opt/softwares [hadoop@clouderaManager01 softwares]$ ll total 4 drwxr-xr-x 2 hadoop hadoop 4096 Apr 8 13:49 cm-5.3.6 [hadoop@clouderaManager01 softwares]$ sudo mkdir cdh-5.3.6 [hadoop@clouderaManager01 softwares]$ ll total 8 drwxr-xr-x 2 root root 4096 Apr 8 20:00 cdh-5.3.6 drwxr-xr-x 2 hadoop hadoop 4096 Apr 8 13:49 cm-5.3.6 [hadoop@clouderaManager01 softwares]$ cd cdh-5.3.6/ [hadoop@clouderaManager01 cdh-5.3.6]$ pwd /opt/softwares/cdh-5.3.6 [hadoop@clouderaManager01 cdh-5.3.6]$ ll total 0 [hadoop@clouderaManager01 cdh-5.3.6]$ sudo rz [hadoop@clouderaManager01 cdh-5.3.6]$ sudo rz [hadoop@clouderaManager01 cdh-5.3.6]$ sudo rz [hadoop@clouderaManager01 cdh-5.3.6]$ ll total 1473900 -rw-r--r-- 1 root root 1509217191 Apr 7 19:38 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html -rw-r--r-- 1 root root 41 Apr 7 19:24 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 -rw-r--r-- 1 root root 42661 Apr 7 19:24 manifest.json [hadoop@clouderaManager01 cdh-5.3.6]$
以下是clouderamanager01这台机器上安装之后,默认产生的(在这里,你看看就好)
[hadoop@clouderaManager01 opt]$ pwd /opt [hadoop@clouderaManager01 opt]$ ll total 16 drwxr-xr-x 4 cloudera-scm cloudera-scm 4096 Apr 8 18:31 cloudera drwxr-xr-x 2 root root 4096 Apr 8 18:13 cloudera-manager drwxr-xr-x. 2 root root 4096 Nov 22 2013 rh drwxr-xr-x 4 hadoop hadoop 4096 Apr 8 20:00 softwares [hadoop@clouderaManager01 opt]$ cd cloudera [hadoop@clouderaManager01 cloudera]$ ll total 8 drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jul 30 2015 csd drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jul 30 2015 parcel-repo [hadoop@clouderaManager01 cloudera]$ cd parcel-repo/ [hadoop@clouderaManager01 parcel-repo]$ pwd /opt/cloudera/parcel-repo [hadoop@clouderaManager01 parcel-repo]$ ll total 0 [hadoop@clouderaManager01 parcel-repo]$
或者,有时候,安装CM完成后,出现:
以下是clouderamanager01这台机器上安装之后,默认产生的(在这里,你看看就好)
[hadoop@clouderamanager01 opt]$ pwd /opt [hadoop@clouderamanager01 opt]$ ll total 8 drwxr-xr-x 4 cloudera-scm cloudera-scm 4096 Apr 8 22:48 cloudera drwxr-xr-x. 3 hadoop hadoop 4096 Apr 8 11:20 softwares [hadoop@clouderamanager01 opt]$ cd cloudera/ [hadoop@clouderamanager01 cloudera]$ ll total 8 drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 csd drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 parcel-repo [hadoop@clouderamanager01 cloudera]$
上传我们刚之前下载好的CDH离线包
[hadoop@clouderamanager01 softwares]$ pwd /opt/softwares [hadoop@clouderamanager01 softwares]$ ll total 4 drwxr-xr-x. 2 hadoop hadoop 4096 Apr 8 11:20 cm-5.3.6 [hadoop@clouderamanager01 softwares]$ sudo mkdir cdh-5.3.6 [hadoop@clouderamanager01 softwares]$ ll total 8 drwxr-xr-x 2 root root 4096 Apr 8 23:04 cdh-5.3.6 drwxr-xr-x. 2 hadoop hadoop 4096 Apr 8 11:20 cm-5.3.6 [hadoop@clouderamanager01 softwares]$ cd cdh-5.3.6/ [hadoop@clouderamanager01 cdh-5.3.6]$ pwd /opt/softwares/cdh-5.3.6 [hadoop@clouderamanager01 cdh-5.3.6]$ ll total 0 [hadoop@clouderamanager01 cdh-5.3.6]$ sudo rz [hadoop@clouderamanager01 cdh-5.3.6]$ sudo rz [hadoop@clouderamanager01 cdh-5.3.6]$ sudo rz [hadoop@clouderamanager01 cdh-5.3.6]$ ll total 1473900 -rw-r--r-- 1 root root 1509217191 Apr 8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html -rw-r--r-- 1 root root 41 Apr 8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 -rw-r--r-- 1 root root 42661 Apr 8 12:33 manifest.json [hadoop@clouderamanager01 cdh-5.3.6]$
注意,是,CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel。而不是CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html,也不是CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.parcel!!!(这个离线包很容易在下载的时候,自己出现后缀的变名。)(若是你下载的时候,没出现变名是更好啦!)
需要改CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1的名称和CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html的名称
[hadoop@clouderamanager01 cdh-5.3.6]$ pwd /opt/softwares/cdh-5.3.6 [hadoop@clouderamanager01 cdh-5.3.6]$ ll total 1473900 -rw-r--r-- 1 root root 1509217191 Apr 8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html -rw-r--r-- 1 root root 41 Apr 8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 -rw-r--r-- 1 root root 42661 Apr 8 12:33 manifest.json [hadoop@clouderamanager01 cdh-5.3.6]$ sudo mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha1 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha [hadoop@clouderamanager01 cdh-5.3.6]$ sudo mv CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.html CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel [hadoop@clouderamanager01 cdh-5.3.6]$ ll total 1473900 -rw-r--r-- 1 root root 1509217191 Apr 8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel -rw-r--r-- 1 root root 41 Apr 8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha -rw-r--r-- 1 root root 42661 Apr 8 12:33 manifest.json [hadoop@clouderamanager01 cdh-5.3.6]$
一定要改成,CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel和CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha。
[hadoop@clouderamanager01 cdh-5.3.6]$ pwd /opt/softwares/cdh-5.3.6 [hadoop@clouderamanager01 cdh-5.3.6]$ ll total 1473900 -rw-r--r-- 1 root root 1509217191 Apr 8 12:47 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel -rw-r--r-- 1 root root 41 Apr 8 12:32 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha -rw-r--r-- 1 root root 42661 Apr 8 12:33 manifest.json [hadoop@clouderamanager01 cdh-5.3.6]$ more CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha 9b0c758c15792be3d7da7597b376668c09a32e2d [hadoop@clouderamanager01 cdh-5.3.6]$
查看manifest.json
http://archive.cloudera.com/cdh5/parcels/5.3.6/manifest.json
在clouderamanager02和clouderamanager03机器上,进入/etc/yum.repos.d/
[hadoop@clouderamanager02 ~]$ cd /etc/yum.repos.d/ [hadoop@clouderamanager02 yum.repos.d]$ pwd /etc/yum.repos.d [hadoop@clouderamanager02 yum.repos.d]$ ll total 16 -rw-r--r--. 1 root root 1926 Nov 27 2013 CentOS-Base.repo -rw-r--r--. 1 root root 638 Nov 27 2013 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 630 Nov 27 2013 CentOS-Media.repo -rw-r--r--. 1 root root 3664 Nov 27 2013 CentOS-Vault.repo [hadoop@clouderamanager02 yum.repos.d]$ sudo vi cloudera-manager.repo
[cloudera-manager] name=Cloudera Manager baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/ gpgcheck=0
或者
[cloudera-manager] name=Cloudera Manager baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/ gpgcheck=1
同理,机器clouderamanager03也要去做。
关于/opt/cloudera/parcel-repo和/opt/cloudera/parcel(必须弄懂,别怕)
192.168.80.148 clouderamanager01 (部署ClouderaManager-server和Mirror server) (部署Agent) (分配1G,我这里只是体验过程,当然你可以分配更多或更小)
192.168.80.149 clouderamanager02 (部署Agent) (分配1G,我这里只是体验过程,当然你可以分配更多或更小)
192.168.80.150 clouderamanager03 (部署Agent)
部署ClouderaManager-server和Mirror server的机器,默认是存储在/opt/cloudera/parcel-repo。
部署Agent的机器,默认是存储在/opt/cloudera/parcel。
即,这也是为什么,我在下面这篇博客里写到。
Cloudera Manager安装之Cloudera Manager 5.3.X安装(三)(tar方式、rpm方式和yum方式)
博客里,提及到的tar方式来安装。
进入,CM安装的机器,自动生成的默认/opt/cloudera/parcel-repo目录下(在clouderamanager01主节点)
[hadoop@clouderamanager01 opt]$ pwd /opt [hadoop@clouderamanager01 opt]$ ll total 8 drwxr-xr-x 4 cloudera-scm cloudera-scm 4096 Apr 8 22:48 cloudera drwxr-xr-x. 4 hadoop hadoop 4096 Apr 8 23:04 softwares [hadoop@clouderamanager01 opt]$ cd cloudera/ [hadoop@clouderamanager01 cloudera]$ pwd /opt/cloudera [hadoop@clouderamanager01 cloudera]$ ll total 8 drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 csd drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 parcel-repo [hadoop@clouderamanager01 cloudera]$ cd parcel-repo/ [hadoop@clouderamanager01 parcel-repo]$ pwd /opt/cloudera/parcel-repo [hadoop@clouderamanager01 parcel-repo]$ ll total 0 [hadoop@clouderamanager01 parcel-repo]$
将我们之前上传在/opt/softwares/cdh-5.3.6(其实这个用作为备份,万一以后丢失了再可以重新安装。免得再下载)的CDH离线安装包,
移动到这个/opt/cloudera/parcel-repo目录上去。
[hadoop@clouderamanager01 parcel-repo]$ pwd /opt/cloudera/parcel-repo [hadoop@clouderamanager01 parcel-repo]$ ll total 0 [hadoop@clouderamanager01 parcel-repo]$ sudo cp /opt/softwares/cdh-5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel . [hadoop@clouderamanager01 parcel-repo]$ sudo cp /opt/softwares/cdh-5.3.6/CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha . [hadoop@clouderamanager01 parcel-repo]$ ll total 1473856 -rw-r--r-- 1 root root 1509217191 Apr 8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel -rw-r--r-- 1 root root 41 Apr 8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha [hadoop@clouderamanager01 parcel-repo]$ sudo cp /opt/softwares/cdh-5.3.6/manifest.json . [hadoop@clouderamanager01 parcel-repo]$ ll total 1473900 -rw-r--r-- 1 root root 1509217191 Apr 8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel -rw-r--r-- 1 root root 41 Apr 8 23:35 CDH-5.3.6-1.cdh5.3.6.p0.11-el6.parcel.sha -rw-r--r-- 1 root root 42661 Apr 8 23:35 manifest.json [hadoop@clouderamanager01 parcel-repo]$
[hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server status cloudera-scm-server (pid 1888) is running... [hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server restart Stopping cloudera-scm-server: [ OK ] Starting cloudera-scm-server: [ OK ] [hadoop@clouderamanager01 parcel-repo]$
等上几分钟,
如果Clouder Manager的7180端口,没起来,怎么办?
[hadoop@clouderamanager01 parcel-repo]$ sudo netstat -tnlp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 901/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 985/master tcp 0 0 0.0.0.0:7432 0.0.0.0:* LISTEN 1850/postgres tcp 0 0 :::80 :::* LISTEN 995/httpd tcp 0 0 :::22 :::* LISTEN 901/sshd tcp 0 0 ::1:25 :::* LISTEN 985/master tcp 0 0 :::7432 :::* LISTEN 1850/postgres [hadoop@clouderamanager01 parcel-repo]$ sudo reboot Broadcast message from hadoop@clouderamanager01 (/dev/pts/0) at 23:45 ... The system is going down for reboot NOW! [hadoop@clouderamanager01 parcel-repo]$ Connection closed by foreign host. Disconnected from remote host(clouderamanager01) at 15:46:15. Type `help' to learn how to use Xshell prompt.
或者
改为本地的,进入/etc/yum.repos.d
即
sudo chmod 755 clouder-manager.repo
[hadoop@clouderamanager01 yum.repos.d]$ pwd /etc/yum.repos.d [hadoop@clouderamanager01 yum.repos.d]$ ll total 20 -rw-r--r--. 1 root root 1926 Nov 27 2013 CentOS-Base.repo -rw-r--r--. 1 root root 638 Nov 27 2013 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 630 Nov 27 2013 CentOS-Media.repo -rw-r--r--. 1 root root 3664 Nov 27 2013 CentOS-Vault.repo -rw-r--r-- 1 root root 202 Apr 9 01:59 cloudera-manager.repo [hadoop@clouderamanager01 yum.repos.d]$ sudo chmod 755 cloudera-manager.repo [hadoop@clouderamanager01 yum.repos.d]$ ll total 20 -rw-r--r--. 1 root root 1926 Nov 27 2013 CentOS-Base.repo -rw-r--r--. 1 root root 638 Nov 27 2013 CentOS-Debuginfo.repo -rw-r--r--. 1 root root 630 Nov 27 2013 CentOS-Media.repo -rw-r--r--. 1 root root 3664 Nov 27 2013 CentOS-Vault.repo -rwxr-xr-x 1 root root 202 Apr 9 01:59 cloudera-manager.repo [hadoop@clouderamanager01 yum.repos.d]$
将
[cloudera-manager] name=Cloudera Manager baseurl=http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/ gpgcheck = 1
改为
[cloudera-manager] # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 name=Cloudera Manager baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/ gpgcheck = 1
注意,此时,所有的机器(比如我这里是clouderamanager01、clouderamanager02和clouderamanager03)都要更改为
[cloudera-manager] # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 name=Cloudera Manager baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
enabled =1 gpgcheck = 1
于是,将所有机器的/etc/yum.repos.d/clouder-manager.repo改好为本地的之后,再次刷新就好
clouderamanager[01-03]
有时候会是如下,
这是,因为。我之前前面3台机器都没有启动服务,所以会是如下。三个自动会勾选。
http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
在安装过程中,出现如下问题
[hadoop@clouderaManager01 html]$ pwd /var/www/html [hadoop@clouderaManager01 html]$ ll total 4 drwxr-xr-x 3 root root 4096 Apr 8 20:37 cm5 [hadoop@clouderaManager01 html]$ sudo mkdir -p cm5/redhat/6/x86_64/cm/5.3.6/repodata [hadoop@clouderaManager01 html]$ ll total 4 drwxr-xr-x 3 root root 4096 Apr 8 20:37 cm5 [hadoop@clouderaManager01 html]$ cd cm5/redhat/6/x86_64/cm/5.3.6/repodata/ [hadoop@clouderaManager01 repodata]$ pwd /var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/repodata [hadoop@clouderaManager01 repodata]$ ll total 0 [hadoop@clouderaManager01 repodata]$
[hadoop@clouderamanager01 repodata]$ pwd /var/www/html/cm5/redhat/6/x86_64/cm/5.3.6/repodata [hadoop@clouderamanager01 repodata]$ lll -bash: lll: command not found [hadoop@clouderamanager01 repodata]$ ll total 0 [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ sudo rz [hadoop@clouderamanager01 repodata]$ ll total 124 -rw-r--r-- 1 root root 91122 Apr 7 22:36 filelists.xml.gz -rw-r--r-- 1 root root 198 Apr 7 22:36 filelists.xml.gz.asc -rw-r--r-- 1 root root 570 Apr 7 22:36 other.xml.gz -rw-r--r-- 1 root root 198 Apr 7 22:36 other.xml.gz.asc -rw-r--r-- 1 root root 4208 Apr 7 22:37 primary.xml.gz -rw-r--r-- 1 root root 198 Apr 7 22:37 primary.xml.gz.asc -rw-r--r-- 1 root root 951 Apr 7 22:37 repomd.xml -rw-r--r-- 1 root root 198 Apr 7 22:37 repomd.xml.asc [hadoop@clouderamanager01 repodata]$
如果还是没有得到解决的话,则
[hadoop@clouderamanager01 yum.repos.d]$ sudo vi cloudera-manager.repo
[cloudera-manager] # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 name=Cloudera Manager baseurl=http://clouderamanager01/cm5/redhat/6/x86_64/cm/5.3.6/
enabled =1 gpgcheck = 1
这里。所有的节点都要一样的。如果其下有其他的,删除(比如cloudera-manager.repo.~!或其他的cloudera-manager.repo.repnew等)
以及重启下
[hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server status cloudera-scm-server (pid 1888) is running... [hadoop@clouderamanager01 parcel-repo]$ sudo service cloudera-scm-server restart Stopping cloudera-scm-server: [ OK ] Starting cloudera-scm-server: [ OK ] [hadoop@clouderamanager01 parcel-repo]$
等上几分钟。
如果还是没有解决的话,说明那个文件cloudera-manager-agent-5.3.6-1.cm536.p0.244.el6.x86_64.rpm已经损坏了。重新下载它,再上传。
然后,再重启
sudo service cloudera-scm-server restart
就可以解决问题了。
如果还是没解决的话,那就换成在线的吧!(哎)
[cloudera-manager] # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 name=Cloudera Manager baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.3.6/ gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera gpgcheck = 1
然后,接着下面
重复,执行上面的过程。这里省略,.....
直至如下界面.....
如果你的网速慢的话,则会出现,
比如我还遇到过这样的一种情况,也就是说,clouderamanager01先安装成功,然后再次执行上面的重头开始,再来clouderamanager02和clouderamanager03安装成功!
安装Parcel到所有主机上(我这里是clouderaManager01、clouderaManager02和clouderaManager03这台机器上)
因为Parcel文件已经提前下载好了,第一个进度条会直接显示已下载,然后开始分配。这个过程实际上就是将Parcel内的软件解压缩安装到各个主机上的过程。耐心等待,时间会比较长,我安装3台主机(即在clouderaManager01、02和03上,安装3节点集群)大概是需要30分钟时间。(这里看个人的网速)
其实啊,这里分配就是解压。
主机检查
检查就可以了,建议不要跳过。静静地等一分钟多点的时间
检查结果如下:
http://clouderamanager01:7180/cmf/express-wizard/wizard#step=hostInspectorStep
因为我这里,之前是在clouderaManager01、clouderaManager02、clouderaManager03和clouderaManager04安装过。
然后,后面,再clouderamanager01、clouderamanager02和clouderamanager03。
所以,会已经有了记录的痕迹。(当然,若是你第一次安装的话,就不会有这个问题出现的哈!)
Cloudera Manager安装之Cloudera Manager安装前准备(CentOS6.5)(一)
永久修改,我们直接修改配置文件,我们直接修改/etc/sysconfig/network
NETWORKING=yes HOSTNAME=clouderaManager01
对应三台都要修改。我这里是自己,当时失误!主机名弄错大小写字母M了。
然后
sudo reboot
这个问题怎么解决呢?
说白了swap就是交换区,当内存容不下这些数据时,就移动到磁盘那去。由于我们是大数据,不希望移动到磁盘。
[hadoop@clouderaManager01 ~]$ sudo free -m total used free shared buffers cached Mem: 980 917 63 0 1 32 -/+ buffers/cache: 883 96 Swap: 1999 552 1447 [hadoop@clouderaManager01 ~]$ sudo cat /proc/sys/vm/swappiness 60 [hadoop@clouderaManager01 ~]$ sudo sysctl -w vm.swappiness=10 vm.swappiness = 10 [hadoop@clouderaManager01 ~]$
clouderaManager01和clouderaManager02和clouderaManager03都要执行。
这个怎么来解决呢?是我们的Linux版本的bug问题。
[hadoop@clouderaManager01 ~]$ su root Password: [root@clouderaManager01 hadoop]# echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag [root@clouderaManager01 hadoop]# vi /etc/rc.local
#!/bin/sh # # This script will be executed *after* all the other init scripts. # You can put your own initialization stuff in here if you don't # want to do the full Sys V style init stuff. touch /var/lock/subsys/local echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
clouderaManager01和clouderaManager02和clouderaManager03都要执行。
带你认识下安装成功之后的目录结构
[root@clouderaManager01 cloudera]# pwd /opt/cloudera [root@clouderaManager01 cloudera]# ll total 16 drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Mar 20 11:24 csd drwxr-xr-x 2 root root 4096 Apr 9 04:14 parcel-cache drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Apr 8 23:52 parcel-repo drwxr-xr-x 4 root root 4096 Apr 9 04:33 parcels [root@clouderaManager01 cloudera]# cd parcels/ [root@clouderaManager01 parcels]# pwd /opt/cloudera/parcels [root@clouderaManager01 parcels]# ll total 4 lrwxrwxrwx 1 root root 26 Apr 9 04:33 CDH -> CDH-5.3.6-1.cdh5.3.6.p0.11 drwxr-xr-x 11 root root 4096 Jul 29 2015 CDH-5.3.6-1.cdh5.3.6.p0.11 [root@clouderaManager01 parcels]# cd CDH [root@clouderaManager01 CDH]# pwd /opt/cloudera/parcels/CDH [root@clouderaManager01 CDH]# ls bin etc include jars lib lib64 libexec meta share [root@clouderaManager01 CDH]#
[root@clouderaManager01 CDH]# pwd /opt/cloudera/parcels/CDH [root@clouderaManager01 CDH]# ll total 80 drwxr-xr-x 2 root root 4096 Jul 29 2015 bin drwxr-xr-x 26 root root 4096 Jul 29 2015 etc drwxr-xr-x 4 root root 4096 Jul 29 2015 include drwxr-xr-x 2 root root 49152 Jul 29 2015 jars drwxr-xr-x 36 root root 4096 Jul 29 2015 lib drwxr-xr-x 2 root root 4096 Jul 29 2015 lib64 drwxr-xr-x 3 root root 4096 Jul 29 2015 libexec drwxr-xr-x 2 root root 4096 Jul 29 2015 meta drwxr-xr-x 3 root root 4096 Jul 29 2015 share [root@clouderaManager01 CDH]# cd bin/ [root@clouderaManager01 bin]# ls avro-tools hadoop hcat kite-dataset oozie solrctl sqoop2-server sqoop-help sqoop-merge zookeeper-client beeline hadoop-0.20 hdfs llama oozie-setup spark-executor sqoop2-tool sqoop-import sqoop-metastore zookeeper-server catalogd hadoop-fuse-dfs hive llamaadmin parquet-tools spark-shell sqoop-codegen sqoop-import-all-tables sqoop-version zookeeper-server-cleanup cli_mt hadoop-fuse-dfs.orig hiveserver2 load_gen pig spark-submit sqoop-create-hive-table sqoop-job statestored zookeeper-server-initialize cli_st hbase impalad mahout pyspark sqoop sqoop-eval sqoop-list-databases whirr flume-ng hbase-indexer impala-shell mapred sentry sqoop2 sqoop-export sqoop-list-tables yarn [root@clouderaManager01 bin]#
安装完成之后,再安装所需的服务(其实,这一步,若你的内存和空间足够,最好选择所有服务)
其实啊,这步是选择集群中要安装的CDH服务,为了将来新增服务麻烦,我们这里选择所有服务。其中有暂时用不到的服务,我们可以在界面中停止服务,这样可以节省内存资源
这里,添加Cloudera Management Service服务
然后,关于安装其他的,如Zookeeper、Hive、HBase和Spakr、Hadoop等。这些。我就不赘述了。
因为,这是,我在自己的电脑虚拟机里搭建的。过卡和内存不够,以下不贴图和不写了。
安装完毕后点继续
贴图,这里不写了。
角色分配
分配各个节点的角色
数据库设置
CDH默认内嵌了一个数据库,如果你需要MySQL或者Oracle请选择使用自定义数据库,
我这里,先点击测试连接,即选择CDH默认的内嵌数据库成功后点继续。
审核更改
这里全部默认就可以了,直接继续。
安装完成,初始化
到这里集群已经安装部署完成了,CDH会自动对你选择的服务进行初始化设置,你也可以稍后自行开启。
初始化完成后,将进入主页面
常见错误
[hadoop@clouderaManager01 parcel-repo]$ sudo service cloudera-scm-server restart ./cloudera-scm-server: line 109: pstree: command not found Starting cloudera-scm-server: [ OK ]
主要因为没有安装pstree包。
解决办法:
[hadoop@clouderaManager01 parcel-repo]$ sudo yum install epel-release (安装yum 源)
[hadoop@clouderaManager01 parcel-repo]$ sudo yum provides pstree (找到提供pstree命令的包)
[hadoop@clouderaManager01 parcel-repo]$ sudo yum install psmisc (安装对应的包)
然后,
[hadoop@clouderaManager01 parcel-repo]$ sudo service cloudera-scm-server restart
Stopping cloudera-scm-server: [ OK ]
Starting cloudera-scm-server: [ OK ]
Detecting Cloudera Manager Server... Detecting Cloudera Manager Server... BEGIN host -t PTR 192.168.1.198 198.1.168.192.in-addr.arpa domain name pointer localhost. END (0) using localhost as scm server hostname BEGIN which python /usr/bin/python END (0) BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' localhost 7182 Traceback (most recent call last): File "<string>", line 1, in <module> File "<string>", line 1, in connect socket.error: [Errno 111] Connection refused END (1) could not contact scm server at localhost:7182, giving up waiting for rollback request
解决办法:
在失败的主机上,终端执行
sudo mv /usr/bin/host /usr/bin/host.bak
然后重试。
安装过程一直卡在"正在获取安装锁"状态界面
解决办法:
在卡住的主机上,删除/tmp文件夹中的所有scm_prepare_node.开头的文件,清空yum缓存,然后重启主机。
sudo cd /tmp sudo rm -rf scm_prepare_node.* sudo yum clean all sudo reboot
安装完毕后NameNode格式化失败,报以下错误
IncorrectVersionException: Unexpected version of storage directory /data/dfs
解决办法:
删除NameNode节点的/data/dfs文件夹,重新添加主机。
如何更改节点ip
解决办法:
重新设置每台主机的hosts文件,完成后进入主机界面,点击重新运行升级向导,再走一遍之前的流程即可。
参考链接
http://www.mamicode.com/info-detail-1596233.html
http://www.jianshu.com/p/57179e03795f
http://www.aboutyun.com/thread-20741-1-1.html
扩展
参考
Cloudera Manager安装之利用parcels方式(在线或离线)安装3或4节点集群(包含最新稳定版本或指定版本的安装)(添加服务)(Ubuntu14.04)(五)
最好,在这之前,先将
注意,在此之前。
cmbigdata1机器上,则需要执行
bigdata@cmbigdata1:~$ sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm scm_password [sudo] password for bigdata: Sorry, try again. [sudo] password for bigdata: JAVA_HOME=/usr/local/jdk/jdk1.8.0_60
Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.7.0_80/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly! bigdata@ubuntucmbigdata1:~$ sudo service cloudera-scm-server start cloudera-scm-server is already running bigdata@cmbigdata1:~$
bigdata@cmbigdata1:~$ sudo service cloudera-scm-agent start
cloudera-scm-agent is already running
bigdata@cmbigdata1:~$
然后,在cmbigdata2、cmbigdata3、cmbigdata4上
bigdata@cmbigdata2:~$ sudo service cloudera-scm-agent start cloudera-scm-agent is already running bigdata@cmbigdata1:~$ bigdata@cmbigdata3:~$ sudo service cloudera-scm-agent start cloudera-scm-agent is already running bigdata@cmbigdata1:~$ bigdata@cmbigdata4:~$ sudo service cloudera-scm-agent start cloudera-scm-agent is already running bigdata@cmbigdata1:~$
因为,我的集群情况是如下
这是,因为。我之前前面4台机器都没有启动服务,所以会是如下。四个自动会勾选。
http://cmbigdata1/cm5/redhat/6/x86_64/cm/5.3.6/
http://cmbigdata1:7180/cmf/express-wizard/resume#step=installStep
对于,第一个黄色警告,其实就是时间同步 的问题。
[bigdata@cmbigdata1 ~]$ ntpdate pool.ntp.org [bigdata@cmbigdata2 ~]$ ntpdate pool.ntp.org [bigdata@cmbigdata3 ~]$ ntpdate pool.ntp.org [bigdata@cmbigdata4 ~]$ ntpdate pool.ntp.org
对于,第二个黄色警告的解决办法!
Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。当前设置为 60。使用 sysctl 命令在运行时更改该设置并编辑 /etc/sysctl.conf 以在重启后保存该设置。您可以继续进行安装,但可能会遇到问题,Cloudera Manager 报告您的主机由于交换运行状况不佳。以下主机受到影响: cmbigdata[1-4]
[bigdata@cmbigdata1 ~]$ su root Password: [root@cmbigdata1 bigdata]# echo 0 > /proc/sys/vm/swappiness [root@cmbigdata1 bigdata]#
然后,接下来,这一步,其实跟我另外写的一篇用ambari是一样的。
Ambari安装之部署3个节点的HA分布式集群
最后,我的初步安装是
因为,在以后,自己是可以修改的。
所以,CDH默认内嵌了一个数据库,无法通过。我这里选择需要MySQL或者Oracle请选择使用自定义数据库,
比如Hive, oozie 元数据库配置,使用我们安装的mysql来存放元数据,其中数据库名,用户名和密码需要预先创建好的。这里。请移步
如果你是一名有经验大数据工程师,无论是安装jdk(注意:CentOS6.5有自带的jdk),还是安装mysql。都是先查看系统是否已经安装Mysql包。
[root@master app]# rpm -qa|grep mysql mysql-libs-5.1.71-1.el6.x86_64 [root@master app]# rpm -e --nodeps mysql-libs-5.1.71-1.el6.x86_64 [root@master app]# rpm -qa|grep mysql [root@master app]#
[root@cmbigdata1 ~]# yum install mysql-server
Is this ok [y/N]: y
[root@cmbigdata1 ~]# service mysqld start
[root@cmbigdata1 ~]# yum install mysql-connector-java Is this ok [y/N]: y
[root@cmbigdata ~]# mysql -u root -p Enter password: //默认密码为空,输入后回车即可 Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 3 Server version: 5.1.73 Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. mysql> set password for root@localhost=password('rootroot'); 密码设置为rootroot 默认情况下Mysql只允许本地登录,所以只需配置root@localhost就好
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
Bye
[root@cmbigdata1 ~]#
[root@cmbigdata1 ~]# mysql -uroot -prootroot
我这里,是第一次安装。所以直接进入
mysql> create database hive DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive'; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> create database clouder_manager DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on clouder_manager.* TO 'clouder_manager'@'%' IDENTIFIED BY 'clouder_manager'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> create database amon DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'amon_password'; Query OK, 0 rows affected (0.00 sec) mysql>
mysql> create database smon DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on smon.* TO 'smon'@'%' IDENTIFIED BY 'smon_password'; Query OK, 0 rows affected (0.00 sec)
mysql> create database rman DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on rman.* TO 'rman'@'%' IDENTIFIED BY 'rman_password'; Query OK, 0 rows affected (0.00 sec)
mysql> create database hmon DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on hmon.* TO 'hmon'@'%' IDENTIFIED BY 'hmon_password'; Query OK, 0 rows affected (0.00 sec)
mysql> create database nav DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on nav.* TO 'hmon'@'%' IDENTIFIED BY 'nav_password'; Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> select user,host,password from mysql.user; +-----------------+------------+-------------------------------------------+ | user | host | password | +-----------------+------------+-------------------------------------------+ | root | localhost | *6C362347EBEAA7DF44F6D34884615A35095E80EB | | root | cmbigdata1 | | | root | 127.0.0.1 | | | | localhost | | | | cmbigdata1 | | | hive | % | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC | | clouder_manager | %t | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 | | amon | % | *4A1243A12C2625553EC0465DD4E640DB056AA07A | | smon | % | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 | | rman | % | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E | | hmon | % | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 | +-----------------+------------+-------------------------------------------+ 11 rows in set (0.00 sec) mysql> exit; Bye [root@cmbigdata1 ~]#
然后,执行,还是如此
参考
Ubuntu 14.04 MySQL 安装配置 For Cloudera Manages 5
解决办法
mysql> grant all on *.* TO 'hive'@'cmbigdata1' IDENTIFIED BY 'hive'; Query OK, 0 rows affected (0.00 sec) mysql> grant all on *.* TO 'amon'@'cmbigdata1' IDENTIFIED BY 'amon_password'; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec)
mysql> grant all on *.* TO 'clouder_manager'@'%cmbigdata1' IDENTIFIED BY 'clouder_manager';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on *.* TO 'smon'@'cmbigdata1' IDENTIFIED BY 'smon_password';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on *.* TO 'rman'@'cmbigdata1' IDENTIFIED BY 'rman_password';
Query OK, 0 rows affected (0.00 sec)
mysql> grant all on *.* TO 'hmon'@'cmbigdata1' IDENTIFIED BY 'hmon_password';
Query OK, 0 rows affected (0.01 sec)
mysql> grant all on *.* TO 'nav'@'cmbigdata1' IDENTIFIED BY 'nav_password';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> select user,host,password from mysql.user;
+-----------------+-------------+-------------------------------------------+
| user | host | password |
+-----------------+-------------+-------------------------------------------+
| root | localhost | *6C362347EBEAA7DF44F6D34884615A35095E80EB |
| root | cmbigdata1 | |
| root | 127.0.0.1 | |
| | localhost | |
| | cmbigdata1 | |
| hive | % | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC |
| clouder_manager | %t | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 |
| amon | % | *4A1243A12C2625553EC0465DD4E640DB056AA07A |
| smon | % | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 |
| rman | % | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E |
| hmon | % | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 |
| hive | cmbigdata1 | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC |
| amon | cmbigdata1 | *4A1243A12C2625553EC0465DD4E640DB056AA07A |
| clouder_manager | %cmbigdata1 | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 |
| smon | cmbigdata1 | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 |
| rman | cmbigdata1 | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E |
| hmon | cmbigdata1 | *DC736DB1383C5AC16BFE895C856F8D30214AF19A |
| nav | cmbigdata1 | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 |
+-----------------+-------------+-------------------------------------------+
18 rows in set (0.00 sec)
mysql>
别忘了oozie
mysql> create database oozie DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.22 sec) mysql> grant all on *.* TO 'oozie'@'cmbigdata1' IDENTIFIED BY 'oozie_password'; Query OK, 0 rows affected (0.18 sec) mysql> grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie_password'; Query OK, 0 rows affected (0.04 sec) mysql> flush privileges; Query OK, 0 rows affected (0.15 sec) mysql> select user,host,password from mysql.user; +-----------------+-------------+-------------------------------------------+ | user | host | password | +-----------------+-------------+-------------------------------------------+ | root | localhost | *6C362347EBEAA7DF44F6D34884615A35095E80EB | | root | cmbigdata1 | | | root | 127.0.0.1 | | | | localhost | | | | cmbigdata1 | | | hive | % | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC | | clouder_manager | %t | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 | | amon | % | *4A1243A12C2625553EC0465DD4E640DB056AA07A | | smon | % | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 | | rman | % | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E | | hmon | % | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 | | hive | cmbigdata1 | *4DF1D66463C18D44E3B001A8FB1BBFBEA13E27FC | | amon | cmbigdata1 | *4A1243A12C2625553EC0465DD4E640DB056AA07A | | clouder_manager | %cmbigdata1 | *B2E4B48ED054F1F571DBCCAB3D08912B73FEF2D5 | | smon | cmbigdata1 | *F65C49868BBCA6D143F5AF81D1864B1C64DF25B4 | | rman | cmbigdata1 | *AEF345BFE495D8E678EA9D3D5708FD110AD2F08E | | hmon | cmbigdata1 | *DC736DB1383C5AC16BFE895C856F8D30214AF19A | | nav | cmbigdata1 | *993A16D875A9AB1932CC2F57E7DADA76AD35C3B2 | | oozie | cmbigdata1 | *81A1BB46F79EBD0AA76E6EFAA31D62458CFCAF62 | | oozie | % | *81A1BB46F79EBD0AA76E6EFAA31D62458CFCAF62 | +-----------------+-------------+-------------------------------------------+ 20 rows in set (0.15 sec) mysql>
下面,这里,是可以以后再更改的。
说白了,这里是启动的总界面,但是呢,我们若这里没启动成功,没关系,可以在后面,单独来启动!(具体如下的详细步骤)
http://cmbigdata1:7180/cmf/clusters/2/express-add-services/index#step=commandDetailsStep
遇到这个问题
Failed to execute command Start on service Key-Value Store Indexer
启动与停止Key-Value Store Indexer
通过,这样,也是可以达到目的的。对于这个问题,以后再来解决!
启动与停止Oozie和相关配置
对于这里,安装出现这个问题,无法解决,后面继续不了 。其实,不然,(可以单独解决)
![](https://i-blog.csdnimg.cn/blog_migrate/69d6a0151be1184fc4b98e2b15040c77.png)
通过,这样,也是可以达到目的的。
启动与停止Hue,并相关配置
启动与停止Cloudera Management Service,并相关配置
http://cmbigdata1:7180/cmf/home
关于安装kafka,见
参考
http://www.doc88.com/p-0042075356791.html