众所周知,星环(Transwarp)于2017年10月底正式对外公开社区版可供社区用户自行下载试用,社区版本与商业版本的主要区别如下图,
下面通过实际步骤与截图介绍如何一步一步搭建TDH集群,仅供参考。关于TDH的系列组件的相关教程,可至Transwarp官网查看详细文档,https://docs.transwarp.io/5.0/ ,不过目前好像未能找到TxSQL相关的文档(可能还未完善)
主要步骤如下,
1 许可证申请
登陆https://snowflake.transwarp.io 并注册用户,之后申请许可证,申请许可证需要提供Server Key,Server Key可在安装完TDH Manager并开始配置集群时获取。获取到许可证后可将许可证另存为本地文件供后续安装集群使用。
2 下载TDH社区版本(TDH-CE)
访问http://transwarp.io/product/tdh_ce 并点击下载,下载可通过网盘下载方式或直接下载方式,网盘下载方式速度较快,建议使用网盘下载方式,下载完的压缩包约5GB。
3 Linux常规配置
包括配置/etc/hosts、关闭iptables、禁用Selinux等,这些步骤也可以在后续通过TDH Manager配置集群时配置。本实验使用4节点虚拟机环境作为测试环境,节点如下。
10.10.22.151 esggy-del-n001.esgyncn.local esggy-del-n001
10.10.22.152 esggy-del-n002.esgyncn.local esggy-del-n002
10.10.22.153 esggy-del-n003.esgyncn.local esggy-del-n003
10.10.22.154 esggy-del-n004.esgyncn.local esggy-del-n004
4 开始安装
通过运行install脚本进行安装,安装首先会下载Oracle JDK1.7,之后会打开8179端口,后续安装便可从http://esggy-del-n001.esgyncn.local:8179/网页进行后续安装。
[root@esggy-del-n001 ~]# tar -xzvf TDH-Basic-Component-Transwarp-ce-1.0.0.tar.gz
[root@esggy-del-n001 ~]# cd transwarp
[root@esggy-del-n001 transwarp]# ./install
current directory: /root/transwarp
preparing Web Installer ...
Loaded plugins: fastestmirror
Examining /root/transwarp/transwarp/hadoop_related/common/jdk-7u71-linux-x64.rpm: 2000:jdk-1.7.0_71-fcs.x86_64
Marking /root/transwarp/transwarp/hadoop_related/common/jdk-7u71-linux-x64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package jdk.x86_64 2000:1.7.0_71-fcs will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===============================================================================================================================================================================================================
Package Arch Version Repository Size
===============================================================================================================================================================================================================
Installing:
jdk x86_64 2000:1.7.0_71-fcs /jdk-7u71-linux-x64 197 M
Transaction Summary
===============================================================================================================================================================================================================
Install 1 Package
Total size: 197 M
Installed size: 197 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 2000:jdk-1.7.0_71-fcs.x86_64 1/1
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
Verifying : 2000:jdk-1.7.0_71-fcs.x86_64 1/1
Installed:
jdk.x86_64 2000:1.7.0_71-fcs
Complete!
executing Web Installer ...
09:22:43.716 [main] DEBUG i.t.manager.installer.util.Shell - cmd: [id, -u] with exit code 0 stdout: 0
stderr:
09:22:43.722 [main] INFO i.t.manager.installer.WebInstaller - staring Web Installer server ...
[I 09:22:45:670] this OS version distro is: CentOS/RHEL 7.x
[I 09:22:45:677] OS release: 7.2
[I 09:22:45:678] prepare result of os-check was NEXT
[I 09:22:45:680] processing os-check
[I 09:22:45:680] process result of os-check was NEXT
[I 09:22:45:684] prepare result of java-license was NEXT
[I 09:22:45:686]
[I 09:22:45:686] The Web Installer is running at http://esggy-del-n001.esgyncn.local:8179/
[I 09:22:45:686] Control-C will stop this server.
[I 09:22:45:686] It is highly recommended that all hosts in the cluster can be visited by FQDN/hostname from the browser. To achieve that, configure proper DNS and/or Hosts file on your browser machine
[I 09:22:45:686]
Copy/paste this URL into your browser when you connect for the first time:
http://esggy-del-n001.esgyncn.local:8179/
……
5 安装TDH Manager
打开http://10.10.22.151:8179/继续执行安装步骤,安装步骤包括:(1)阅读Java许可协议;(2)检查环境;(3)提取文件;(4)配置网络;(5)配置RPM库;(6)安装Manager。具体步骤如下截图,安装完成后会启用8180作为默认的TDH Manager网页端口,后续可通过8180端口继续Hadoop集群的安装。
注:在最后一步安装Manager时遇到以下错误
Volume group "docker" not found
Cannot process volume group docker
Missing specific volume group named 'docker'
解决方法为在每个节点添加一块至少40G的volume group,这时因为新版本的TDH将所有的组件都包装在了docker中,为了更好的资源隔离和性能使用。
yum install -y lvm2;
pvcreate /dev/vdb;
vgcreate docker /dev/vdb
[root@esggy-del-n001 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
docker 2 2 0 wz--n- 39.99g 7.88g
6 安装Hadoop集群
打开8180网页,使用admin/admin登陆TDH Manager,依次执行上传许可证、配置集群名称、添加节点、添加服务等步骤。这些步骤与CDH Manager执行安装十分相似,稍微有所不同,具体步骤如下面截图。
7 附TDH架构图