Win主机+Linux虚拟机搭建Kettle集群

环境说明

程序               版本

Win:OS        win 7

Linux:OS       Redhat 6.2

JDK            1.8.0_45

Kettle         5.1

本机:Vmnet1   192.168.52.1

集群Master：  192.168.52.1

集群Slave1：  192.168.52.128

集群Slave2：  192.168.52.1

需要在Redhat 6.2上安装好jdk，配置好环境变量，并且把kettle安装好。（kettle不需要启动spoon来启动图形界面，利用kitchen调试好即可，linux上部署kettle，解压即可，因为kettle是跨平台运行的）。

注意：在集群运行环境中，需要端口打开或者直接关闭防火墙，供主服务器及子服务器互相连通通讯，在Redhat 6.2中，防火墙采用iptables的方式开启和关闭，检查端口是否开通，可以通过本机进行telnet测试。

首先，配置kettle主目录下的文件夹pwd中的carte-config-master-8080.xml、carte-config-8081.xml和carte-config-8082.xml文件，配置文件见下图：

注意：配置文件中的username和password并不是指主机的登陆账号和密码，是集群的账号密码，该账号密码是集群连接的依据，账号密码是通过混淆的方式保存在pwd文件，kettle默认的账号密码是cluster/cluster，修改该账号密码需要修改pwd文件，该文件也在pwd目录中，密码做了混淆。

Tips：

Kettle的集群搭建还是比较简单的，但是简单也有坑。

坑1：基于主机+虚拟机搭建集群的时候，一定要注意虚拟机与主机的网络通信方式，同时要考虑win主机的hosts文件的内容修改；

坑2：官网下载的kettle5或者6的主目录下是没有安装集群所必须的必要文件，可以去该地址下载相应文件：

Maven Central Repository Search

http://repo1.maven.org/maven2/org/owasp/esapi/esapi/2.1.0.1/

集群必要文件列表：

ESAPI.properties

validation.properties

antisamy-esapi.xml

集群必要jar文件：

esapi-2.1.0.1.jar

这个jar包记得放在lib文件夹里。

下面把配置步骤做如下描述：

一.  MASTER 配置

进到pwd目录，打开文件carte-config-master-8080.xml，内容如下

<slave_config>   <!--      Document description...           - masters: You can list the slave servers to which this slave has to report back to.                 If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.      - report_to_masters : send a message to the defined masters to let them know we exist (Y/N)      - slaveserver : specify the slave server details of this carte instance.                      IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.   -->   <slaveserver>     <name>master1</name>     <hostname>192.168.52.1</hostname>     <port>8080</port>     <master>Y</master>   </slaveserver> </slave_config>

Tips：这里注意<hostname>的value值选择本机的Vmnet1地址，因为我的主机与虚拟机是通过host-only方式通信的，如果你的虚拟机是通过NAT方式通信，那么此时的value要填入Vmnet8地址，原因自己想，理由很简单。

此时MASTER配置完毕。

二.  SLAVE2 配置

进到pwd目录，打开文件carte-config-8082.xml，内容如下

<slave_config>   <!--      Document description...           - masters: You can list the slave servers to which this slave has to report back to.                 If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.      - report_to_masters : send a message to the defined masters to let them know we exist (Y/N)      - slaveserver : specify the slave server details of this carte instance.                      IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.   -->   <masters>     <slaveserver>       <name>master1</name>       <hostname>192.168.52.1</hostname>       <port>8080</port>       <username>cluster</username>       <password>cluster</password>       <master>Y</master>     </slaveserver>   </masters>   <report_to_masters>Y</report_to_masters>   <slaveserver>     <name>slave2-8082</name>     <hostname>192.168.52.1</hostname>     <port>8082</port>     <username>cluster</username>     <password>cluster</password>     <master>N</master>   </slaveserver> </slave_config>

Tips：

子服务器比主服务器的配置内容多一点，该子服务器其实是与MASTER配置在一台主机上，除了自身的配置信息以外，要把MASTER的配置信息也配置好，因为启动子服务器的时候，子服务器需要根据配置文件中的MASTER地址和端口去MASTER主机注册服务，这里要注意的是配置子服务器的地方有个K-V配置，即<master>N</master>，如果是配置MASTER，这个位置的值是Y，如果配置的是子服务器，那么这个位置的值是N。

三.  SLAVE3 配置

进到pwd目录，打开文件carte-config-8081.xml，内容如下

<slave_config>   <!--      Document description...      - masters: You can list the slave servers to which this slave has to report back to.                 If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.      - report_to_masters : send a message to the defined masters to let them know we exist (Y/N)      - slaveserver : specify the slave server details of this carte instance.                      IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.   -->   <masters>     <slaveserver>       <name>master1</name>       <hostname>192.168.52.1</hostname>       <port>8080</port>       <username>cluster</username>       <password>cluster</password>       <master>Y</master>     </slaveserver>   </masters>   <report_to_masters>Y</report_to_masters>   <slaveserver>     <name>slave1-8081</name>     <hostname>192.168.52.128</hostname>     <port>8081</port>     <username>cluster</username>     <password>cluster</password>     <master>N</master>

Tips：slave3是一台linux虚拟机，特别要注意在配置MASTER时的hostname的值，同时还要切记将前文提到的三个必要文件放在kettle的主目录中，否则会报错，报错内容如下：

Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: /opt/data-integration-5.1/validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties Not found in 'user.home' (/root) directory: /root/esapi/validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. validation.properties could not be loaded by any means. fail. Exception was: java.lang.IllegalArgumentException: Failed to load ESAPI.properties as a classloader resource.

启动MASTER主机的carte服务：

D:\software\data-integration-5.1>Carte.bat pwd\carte-config-master-8080.xml DEBUG: Using JAVA_HOME DEBUG: _PENTAHO_JAVA_HOME=D:\jdk1.8.0_74 DEBUG: _PENTAHO_JAVA=D:\jdk1.8.0_74\bin\java.exe 2017/12/27 16:31:33 - Carte - Installing timer to purge stale objects after 1440  minutes. 2017-12-27 16:31:33.331::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog 2017/12/27 16:31:33 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8080 2017-12-27 16:31:33.433::INFO:  jetty-6.1.21 2017-12-27 16:31:33.458::INFO:  Started SocketConnector@192.168.52.1:8080 Attempting to load ESAPI.properties via file I/O. Attempting to load ESAPI.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\ESAPI.properties Not found in SystemResource Directory/resourceDirectory: .esapi\ESAPI.properties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\ESAPI.properties Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun dException Attempting to load ESAPI.properties via the classpath. SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from '/ (root)' using cur rent thread context class loader! Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi\validation.prope rties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. SUCCESSFULLY LOADED validation.properties via the CLASSPATH from '/ (root)' usin g current thread context class loader!

打开浏览器，输入地址：http://192.168.52.1:8080

输入用户名和密码(就是在配置文件中配置的用户名和密码)

Tips:注意右下角，有提示这是MASTER.

启动slave1

[root@racnode2 data-integration-5.1]# carte.sh pwd/carte-config-8081.xml /opt/data-integration-5.1 2017/12/27 00:34:21 - Carte - Installing timer to purge stale objects after 1440 minutes. System property [org.owasp.esapi.opsteam] is not set System property [org.owasp.esapi.devteam] is not set Attempting to load ESAPI.properties via file I/O. Attempting to load ESAPI.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: /opt/data-integration-5.1/ESAPI.properties Not found in SystemResource Directory/resourceDirectory: .esapi/ESAPI.properties Not found in 'user.home' (/root) directory: /root/esapi/ESAPI.properties Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoundException Attempting to load ESAPI.properties via the classpath. SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from '/ (root)' using current thread context class loader! SecurityConfiguration for Validator.ConfigurationFile.MultiValued not found in ESAPI.properties. Using default: false Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: /opt/data-integration-5.1/validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties Not found in 'user.home' (/root) directory: /root/esapi/validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. SUCCESSFULLY LOADED validation.properties via the CLASSPATH from '/ (root)' using current thread context class loader! 2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080] 2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See SLF4J Error Codes for further details. 2017/12/27 00:34:22 - Carte - Created listener for webserver @ address : 192.168.52.128:8081

打开浏览器，输入地址：http://192.168.52.128:8081

启动slave2

D:\software\data-integration-5.1>Carte.bat pwd\carte-config-8082.xml DEBUG: Using JAVA_HOME DEBUG: _PENTAHO_JAVA_HOME=D:\jdk1.8.0_74 DEBUG: _PENTAHO_JAVA=D:\jdk1.8.0_74\bin\java.exe 2017/12/27 16:38:09 - Carte - Installing timer to purge stale objects after 1440  minutes. Attempting to load ESAPI.properties via file I/O. Attempting to load ESAPI.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\ESAPI.properties Not found in SystemResource Directory/resourceDirectory: .esapi\ESAPI.properties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\ESAPI.properties Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun dException Attempting to load ESAPI.properties via the classpath. SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from '/ (root)' using cur rent thread context class loader! Attempting to load validation.properties via file I/O. Attempting to load validation.properties as resource file via file I/O. Not found in 'org.owasp.esapi.resources' directory or file not readable: D:\soft ware\data-integration-5.1\validation.properties Not found in SystemResource Directory/resourceDirectory: .esapi\validation.prope rties Not found in 'user.home' (C:\Users\Administrator) directory: C:\Users\Administra tor\esapi\validation.properties Loading validation.properties via file I/O failed. Attempting to load validation.properties via the classpath. SUCCESSFULLY LOADED validation.properties via the CLASSPATH from '/ (root)' usin g current thread context class loader! 2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve r [master1] on address [192.168.52.1:8080] 2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve r [master1] on address [192.168.52.1:8080] 2017-12-27 16:38:09.398::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog 2017/12/27 16:38:09 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8082 2017-12-27 16:38:09.456::INFO:  jetty-6.1.21 2017-12-27 16:38:09.475::INFO:  Started SocketConnector@192.168.52.1:8082

打开浏览器，输入地址：http://192.168.52.1:8082

关于在kettle开发过程中转换和工作流里的集群配置，以后再写个文章说明。