六、安装部署HDP集群
浏览器访问 http://master:8080,进入amabri登录页面,用户名:admin,密码: admin
选择 Launch Install Wizard:
1. Get started
给集群起个名字,这里是 BigData:
2. Select stack
选择 hdp2.3, 将除redhat6 以外的复选框去掉勾。 并且将hdp以及hdp-utiles的baseurl 替换掉默认的值。
3. Install options
写入三台机器的GQDN,并选择第一项provide SSHPrivate Key ,浏览私钥文件id_rsa.pub,这样部署集群时会在slave1和slave2上安装ambari-agent。如图6.3.1。如果提前安装了amabri-agent可以选第二项。
提前手动安装ambari-agent:
<code class="hljs vala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#yum install ambari-agent</span> <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#vi /etc/ambari-agent/conf/ambari-agent.ini</span> [server] hostname=<your.ambari.server.hostname> url_port=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8440</span> secured_url_port=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8441</span> <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#ambari-agent start</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li></ul>
在三台机器执行上述代码,其中 < your.ambari.server.hostname> ,根据每台机器的hostname来填即可。
点击下一步。
4.Confirm hosts
图6.4.1
图6.4.2
图6.4.3
如图6.4.2 ,有警告信息,点开查看如图6.4.3,提示 关闭 THP,
<code class="hljs vala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;"># vi /etc/grub.conf</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
加入transparent_hugepage=never,,
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">title CentOS (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.32</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">431.17</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.el</span>6<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.x</span>86_64) root (hd0,<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) kernel /vmlinuz-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.32</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">431.17</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.el</span>6<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.x</span>86_64 root=/dev/mapper/VolGroup-lv_root ro rd_NO_LUKS rd_NO_MD rd_LVM_LV=VolGroup/lv_swap LANG=zh_CN<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.UTF</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">8</span> rd_LVM_LV=VolGroup/lv_root transparent_hugepage=never KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /initramfs-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.32</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">431.17</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.el</span>6<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.x</span>86_64<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.img</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
继续,如图6.4.5可以看到没有错误了。
图6.4.5
5.Choose service
服务可根据需要自行选择。
6.Assign masters
根据负载均衡的原则,将要安装的服务分配到各台机器。本集群的组件分配见表2.2.2。提示:各个组件的Master可都安在master机器上。
7.Assign slaves and clients
可根据需要在master、slave1、slave2上安装组件,这里我们选择三台机器上均安装客户端client。
8.Custom services
看一下各项的参数配置,最后看hive。 主要看内存配置,例如 Java heap size 等,根据机器总内存大小来分配,本集群是将内存设成默认值的一半。这些参数待集群建好后都是可以再调整的。
看完其他项,接着该 验证hive 了。
MySQL是安在master上,如图6.8.2填写数据库所在机器的域名,将第五部分提前为hive建立的数据库信息填入下表,点击测试连接。
点击Next:有提示,不用管,是因为我们将Java heap size value调小了。
继续往下,确认集群的各项:
选择deploy,接下来是确认部署信息、启动以及测试了。
Error:可能会出错,截取部分出错信息:
<code class="hljs oxygene has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">Error Flume: resource_management.core.exceptions.Fail: Execution <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">of</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'/usr/bin/yum -d 0 -e 0 -y install '</span>flume_2_3_*<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">''</span> returned <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>. Error: Package: hadoop_2_3_2_0_2950-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.2</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3.2</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2950</span>.el6.x86_64 (HDP-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.3</span>) Requires: nc You could <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">try</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">using</span> --<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">skip</span>-broken <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> work around the problem ** Found <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span> pre-existing rpmdb problem(s), <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">'yum check'</span> output follows: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>:postfix-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>.el6_5.x86_64 <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">has</span> missing requires <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">of</span> libmysqlclient.so.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>()(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">64</span>bit) <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>:postfix-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>.el6_5.x86_64 <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">has</span> missing requires <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">of</span> libmysqlclient.so.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span>(libmysqlclient_16)(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">64</span>bit) <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>:postfix-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span>.el6_5.x86_64 <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">has</span> missing requires <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">of</span> mysql-libs </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li></ul>
解决:
提示缺少nc, 执行yum install nc
也有可能是超时,vi /etc/yum.conf
,将 installonly_limit的值设成600
vi /var/lib/ambari-server/resources/stacks/HDP/2.3/repos/repoinfo.xml
,bseurl 改一下。
retry几次,最后即使没有全部安好,也可先点Deploy部署集群,之后再把没有安成功的服务重新安装即可。
分别点开三台机器,如图可以查看在三台机器上哪些组件安好、哪些没安好。
七、启动服务
集群部署好以后,许多服务是关闭的,因此会有很多警告。
我们先将服务一项项的启动,发现警告也会一点点的消除。可能有些组件没有安成功,点击hosts,点开每台机器,会看到有些组件安装失败,点击再安装即可。
原文来自:http://blog.csdn.net/sinat_28224453/article/details/51538449