DIX系统
此处 DIX 使用仅限于在 CamBuilder 中
DIX介绍
DIX是CamBuilder中用于实时对接入数据进行处理后输出到指定目的地的系统。可以用来接入Mysql、ActiveMq,syslog等数据源中存储的数据,可以在接入这些数据源存储的数据后,利用DIX对数据进行简单处理后,输出到指定的数据源,目前支持MySql、ActiveMq等。
通过访问DIX服务器进入dip配置页面,选择JavaScript,点击立即接入按钮之后进入了dip的配置页面,在该页面进行dip的属性配置、脚本配置。
配置完成上述信息之后,点击完成并保存,一个dip便生成了,然后回到数据集成页面找到对应的dip,启动即可。
![](https://i-blog.csdnimg.cn/blog_migrate/c858fbc2d8271d1fe69792a0f8710d78.png)
数据接入
接收MySQL数据
点击JavaScript,进入配置页面,在1处填入dip名字,2处的地址为宿主机地址,默认为dip系统部署机器地址, 点击下一步,填写脚本配置。
dip名称:使用英文且见名知义(最好使用驼峰命名法)。
dip的宿主机:运行该dip的服务器ip地址,也是我们的数据处理服务器。
![](https://i-blog.csdnimg.cn/blog_migrate/ab9fc83bb0a9de607c8a2cffbac477bc.png)
进入如下界面,在黑窗口的run函数中编写JavaScript脚本,将获取到的数据转换成产品需要的数据结构,最后输出。
首先编写脚本获取数据,获取数据有2种方式,第一种是主动获取,第二种是被动接收,根据实际情况选择对应的获取数据方式。而每一种方式又有多种类型。
![](https://i-blog.csdnimg.cn/blog_migrate/3de44cfebde3d7c6b22456f40bd62e44.png)
![](https://i-blog.csdnimg.cn/blog_migrate/bb32326e2d7f7d697a893a22ca6e4aa8.png)
对接MySQL我们选择主动获取方式,点击输入-主动获取,选择第一项MySQL ,相应的接入MySQL的代码会自动填写在function run() 里面 ,in 为接入语句,executes为数据处理逻辑,outs处填写数据输出语句。
如下图,在1处添加接入MySQL数据源url,username,password,在2处添加查询简单的查询sql语句,查询自己需要的数据。
![](https://i-blog.csdnimg.cn/blog_migrate/452bd6e768ff8f038bbf859ff3e692a9.png)
<span style="color:#1f3349"><span style="color:#273d56 !important"><code class="language-javascript"><span style="color:#314ccd">function</span> <span style="color:#4264fb">run</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> MysqlJdbcAdapter <span style="color:#273d56">=</span> Java<span style="color:#273d56">.</span><span style="color:#4264fb">type</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"com.uinnova.di.dicom.jdbc.MysqlJdbcAdapter"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">while</span> <span style="color:#273d56">(</span><span style="color:#7753eb">true</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">try</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> adapter <span style="color:#273d56">=</span> <span style="color:#314ccd">new</span> <span style="color:#4264fb">MysqlJdbcAdapter</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"jdbc:mysql://127.0.0.1:3306/database?useUnicode=true&characterEncoding=UTF-8"</span><span style="color:#273d56">,</span> <span style="color:#ce2c69">"root"</span><span style="color:#273d56">,</span> <span style="color:#ce2c69">"root"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">var</span> data <span style="color:#273d56">=</span> adapter<span style="color:#273d56">.</span><span style="color:#4264fb">getDataByTable</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"select * from ..."</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
heartBeat<span style="color:#273d56">.</span><span style="color:#4264fb">addInCount</span><span style="color:#273d56">(</span>data<span style="color:#273d56">.</span><span style="color:#4264fb">size</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#53708e">//executes...</span>
<span style="color:#53708e">//outs...</span>
<span style="color:#4264fb">sleep</span><span style="color:#273d56">(</span><span style="color:#7753eb">60000</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#273d56">}</span> <span style="color:#314ccd">catch</span> <span style="color:#273d56">(</span>error<span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
logger<span style="color:#273d56">.</span><span style="color:#4264fb">error</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"脚本执行错误:"</span><span style="color:#273d56">,</span>error<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span></code> </span></span>
接收ActiveMq数据
点击JavaScript,进入配置页面,在1处填入dip名字,2处的地址为宿主机地址,默认为dip系统部署机器地址, 点击下一步,填写脚本配置。
dip名称:使用英文且见名知义(最好使用驼峰命名法)。
dip的宿主机:运行该dip的服务器ip地址,也是我们的数据处理服务器。
![](https://i-blog.csdnimg.cn/blog_migrate/5cd6d74ce8224008d48b58dfc33382e7.png)
进入如下界面,点击输入-被动接受,选择第一项 ,相应的接入ActiveMq 的代码会自动填写在function run() 里面 in 为接入语句,executes为数据处理逻辑,outs处填写数据输出语句。
如下图,在1处依次填入ActiveMq数据源的url,queue,username,pasword,最后一个参数为true,表示接受queue,为false表示接受topic,
URL: MQ的地址,格式为tcp://${ip}:${port},例如tcp://192.168.10.44:61616
对列名queue: 产品接收的指定队列noah_perfs.
用户名username: admin
密码password: admin
![](https://i-blog.csdnimg.cn/blog_migrate/f093fcd6673c6732bd3c4ec6a3414b80.png)
<span style="color:#1f3349"><span style="color:#273d56 !important"><code class="language-javascript"><span style="color:#314ccd">function</span> <span style="color:#4264fb">run</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> ActivemqReceiver <span style="color:#273d56">=</span> Java<span style="color:#273d56">.</span><span style="color:#4264fb">type</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"com.uinnova.di.dip.base.receiver.ActivemqReceiver"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#53708e">//url, queue, username, password, queue(true) or topic(false)</span>
<span style="color:#314ccd">var</span> receiver <span style="color:#273d56">=</span> <span style="color:#314ccd">new</span> <span style="color:#4264fb">ActivemqReceiver</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"tcp://127.0.0.1:61616"</span><span style="color:#273d56">,</span> <span style="color:#ce2c69">"queue"</span><span style="color:#273d56">,</span> <span style="color:#ce2c69">"admin"</span><span style="color:#273d56">,</span> <span style="color:#ce2c69">"admin"</span><span style="color:#273d56">,</span> <span style="color:#7753eb">true</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#4264fb">startReceiver</span><span style="color:#273d56">(</span>receiver<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">while</span> <span style="color:#273d56">(</span><span style="color:#7753eb">true</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">try</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> data <span style="color:#273d56">=</span> receiver<span style="color:#273d56">.</span><span style="color:#4264fb">getData</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">if</span> <span style="color:#273d56">(</span>data<span style="color:#273d56">!=</span><span style="color:#314ccd">null</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
heartBeat<span style="color:#273d56">.</span><span style="color:#4264fb">addInCount</span><span style="color:#273d56">(</span><span style="color:#7753eb">1</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#53708e">//executes...</span>
<span style="color:#53708e">//outs...</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span> <span style="color:#314ccd">catch</span> <span style="color:#273d56">(</span>error<span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
logger<span style="color:#273d56">.</span><span style="color:#4264fb">error</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"脚本执行错误:"</span><span style="color:#273d56">,</span>error<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span></code> </span></span>
接收Syslog数据
点击JavaScript,进入配置页面,在1处填入dip名字,2处的地址为宿主机地址,默认为dip系统部署机器地址, 点击下一步,填写脚本配置。
dip名称:使用英文且见名知义(最好使用驼峰命名法)。
dip的宿主机:运行该dip的服务器ip地址,也是我们的数据处理服务器。
![](https://i-blog.csdnimg.cn/blog_migrate/bcbe83a9a1a102e80384b3f250bfb122.png)
进入如下界面,在1处点击输入-被动接受,选择第一项 ,相应的接入Syslog 的代码会自动填写在function run() 里面 in 为接入语句,executes为数据处理逻辑,outs处填写数据输出语句。
IN配置是指接收syslog告警数据必须的一些信息,根据具体的需求调研填写
依次填入Syslog数据源的url,port,protocal,
主机url:部署DIP服务端程序的主机IP
端口port:与第三方厂商syslog确认的syslog数据推送端口,建议填写大于1600的未被其他服务占用的端口,
协议protocal:与第三方厂商syslog确认进行syslog数据传输协议,可选UDP或TCP
接入syslog数据库
![](https://i-blog.csdnimg.cn/blog_migrate/54248abaece90cce6646e52b46edc7ab.png)
<span style="color:#1f3349"><span style="color:#273d56 !important"><code class="language-javascript"><span style="color:#314ccd">function</span> <span style="color:#4264fb">run</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> SyslogReceiver <span style="color:#273d56">=</span> Java<span style="color:#273d56">.</span><span style="color:#4264fb">type</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"com.uinnova.di.dip.base.receiver.SyslogReceiver"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#53708e">//host, port, protocol(TCP or UDP)</span>
<span style="color:#314ccd">var</span> receiver <span style="color:#273d56">=</span> <span style="color:#314ccd">new</span> <span style="color:#4264fb">SyslogReceiver</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"0.0.0.0"</span><span style="color:#273d56">,</span> <span style="color:#7753eb">1501</span><span style="color:#273d56">,</span> <span style="color:#ce2c69">"UDP"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#4264fb">startReceiver</span><span style="color:#273d56">(</span>receiver<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">while</span> <span style="color:#273d56">(</span><span style="color:#7753eb">true</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">try</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> data <span style="color:#273d56">=</span> receiver<span style="color:#273d56">.</span><span style="color:#4264fb">getData</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
logger<span style="color:#273d56">.</span><span style="color:#4264fb">info</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"[原始数据]"</span> <span style="color:#273d56">+</span> data<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">if</span> <span style="color:#273d56">(</span>data <span style="color:#273d56">!=</span> <span style="color:#314ccd">null</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
heartBeat<span style="color:#273d56">.</span><span style="color:#4264fb">addInCount</span><span style="color:#273d56">(</span><span style="color:#7753eb">1</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#53708e">//executes...</span>
<span style="color:#53708e">//outs...</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span> <span style="color:#314ccd">catch</span> <span style="color:#273d56">(</span>error<span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
logger<span style="color:#273d56">.</span><span style="color:#4264fb">error</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"脚本执行错误:"</span><span style="color:#273d56">,</span> error<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span>
<span style="color:#273d56">}</span></code> </span></span>
数据处理
处理MySQL数据
我们可以取出MySQL中的数据进行处理,将他们转为我们需要的格式,在进行输出。
在编写脚本数据转换的时候可以点击数据处理, 就会出现如下图所示的子菜单,可以根据接收到的数据类型(告警/性能)选择对应的菜单,脚本框中会出现对应的模板代码,如下图所示。
![](https://i-blog.csdnimg.cn/blog_migrate/63a7b166a2a24cc1ff8d0cf4cc5a248f.png)
以JavaScript处理接入MySQL为例,准备数据接入。
样例数据:
<span style="color:#1f3349"><span style="color:#273d56 !important"><code class="language-json">[
{
"id": "1",
"sceneid": "20170109153114611633853",
"kpi_key": "temperature",
"kpi_value": "253"
}
]</code> </span></span>
如下图,1处接入MySQL数据,2处对数据进行处理,可以参考下面数据处理中的性能数据映射代码,3处把处理后的数据同样输出到另外一个MySQL中去,对其他数据源的处理可以参照此处进行处理。
数据处理后,输出到MySQL数据。
DIP代码参考如下:
<span style="color:#1f3349"><span style="color:#273d56 !important"><code class="language-javascript"><span style="color:#314ccd">function</span> <span style="color:#4264fb">run</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> MysqlJdbcAdapter <span style="color:#273d56">=</span> Java<span style="color:#273d56">.</span><span style="color:#4264fb">type</span><span style="color:#273d56">(</span>
<span style="color:#ce2c69">"com.uinnova.di.dicom.jdbc.MysqlJdbcAdapter"</span>
<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">while</span> <span style="color:#273d56">(</span><span style="color:#7753eb">true</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">try</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> adapter <span style="color:#273d56">=</span> <span style="color:#314ccd">new</span> <span style="color:#4264fb">MysqlJdbcAdapter</span><span style="color:#273d56">(</span>
<span style="color:#ce2c69">"jdbc:mysql://192.168.137.128:3306/inputDB?useUnicode=true&characterEncoding=UTF-8"</span><span style="color:#273d56">,</span>
<span style="color:#ce2c69">"root"</span><span style="color:#273d56">,</span>
<span style="color:#ce2c69">"root"</span>
<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">var</span> data <span style="color:#273d56">=</span> adapter<span style="color:#273d56">.</span><span style="color:#4264fb">getDataByTable</span><span style="color:#273d56">(</span>
<span style="color:#ce2c69">"SELECT id,sceneid,kpi_key,kpi_value FROM c_in"</span>
<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
heartBeat<span style="color:#273d56">.</span><span style="color:#4264fb">addInCount</span><span style="color:#273d56">(</span>data<span style="color:#273d56">.</span><span style="color:#4264fb">size</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#53708e">//executes...</span>
<span style="color:#53708e">//var javaListData = json2JavaList(data);</span>
<span style="color:#314ccd">var</span> list <span style="color:#273d56">=</span> <span style="color:#314ccd">new</span> <span style="color:#4264fb">Array</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">for</span> <span style="color:#273d56">(</span><span style="color:#314ccd">var</span> i <span style="color:#273d56">=</span> <span style="color:#7753eb">0</span><span style="color:#273d56">;</span> i <span style="color:#273d56"><</span> data<span style="color:#273d56">.</span><span style="color:#4264fb">size</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span> i<span style="color:#273d56">++</span><span style="color:#273d56">)</span> <span style="color:#273d56">{</span>
<span style="color:#314ccd">var</span> obj <span style="color:#273d56">=</span> <span style="color:#314ccd">new</span> <span style="color:#4264fb">Object</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#314ccd">var</span> temp <span style="color:#273d56">=</span> data<span style="color:#273d56">.</span><span style="color:#4264fb">get</span><span style="color:#273d56">(</span>i<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
obj<span style="color:#273d56">.</span>id <span style="color:#273d56">=</span> temp<span style="color:#273d56">.</span><span style="color:#4264fb">get</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"id"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
obj<span style="color:#273d56">.</span>sceneid <span style="color:#273d56">=</span> temp<span style="color:#273d56">.</span><span style="color:#4264fb">get</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"sceneid"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
obj<span style="color:#273d56">.</span>kpi_key <span style="color:#273d56">=</span> temp<span style="color:#273d56">.</span><span style="color:#4264fb">get</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"kpi_key"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
obj<span style="color:#273d56">.</span>kpi_value <span style="color:#273d56">=</span> temp<span style="color:#273d56">.</span><span style="color:#4264fb">get</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"kpi_value"</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
logger<span style="color:#273d56">.</span><span style="color:#4264fb">info</span><span style="color:#273d56">(</span><span style="color:#ce2c69">"obj处理后数据"</span> <span style="color:#273d56">+</span> obj<span style="color:#273d56">.</span><span style="color:#4264fb">toString</span><span style="color:#273d56">(</span><span style="color:#273d56">)</span><span style="color:#273d56">)</span><span style="color:#273d56">;</span>
list<span style="color:#273d56">.</span><span style="color:#4264fb">push</span><span style="color:#273d56">(</span>obj<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#273d56">}</span>
<span style="color:#53708e">//logger.info("listlength="+list.length);</span>
<span style="color:#53708e">//outs...</span>
<span style="color:#53708e">//下列为操作数据库的几个方法,传入参数sql,执行成功返回1,执行失败返回0</span>
<span style="color:#53708e">//out.createTable(sql);创建表</span>
<span style="color:#53708e">//out.updateTable(sql); 修改表数据</span>
<span style="color:#53708e">//out.deleteTable(sql);删除表</span>
<span style="color:#53708e">//out.updateTableStructure(sql); 修改表结构</span>
<span style="color:#53708e">//out.selectTable(sql); 查询表</span>
<span style="color:#314ccd">var</span> MysqlOutCustom <span style="color:#273d56">=</span> Java<span style="color:#273d56">.</span><span style="color:#4264fb">type</span><span style="color:#273d56">(</span>
<span style="color:#ce2c69">"com.uinnova.di.dip.customoutmysql.MysqlOutCustom"</span>
<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="color:#53708e">//url, username, passowrd</span>
<span style="color:#314ccd">var</span> out <span style="color:#273d56">=</span> <span style="color:#314ccd">new</span> <span style="color:#4264fb">MysqlOutCustom</span><span style="color:#273d56">(</span>
<span style="color:#ce2c69">"jdbc:mysql://192.168.137.138:3306/outputDB?useUnicode=true&characterEncoding=UTF-8"</span><span style="color:#273d56">,</span>
<span style="color:#ce2c69">"root"</span><span style="color:#273d56">,</span>
<span style="color:#ce2c69">"root"</span>
<span style="color:#273d56">)</span><span style="color:#273d56">;</span>
<span style="co