Storm新特性之Flux
Flux是Storm版本0.10.0中的新组件,主要目的是为了方便拓扑的开发与部署。原先在开发Storm拓扑的时候整个拓扑的结构都是硬编码写在代码中的,当要对其进行修改时,需要修改代码并重新编译和打包,这是一件繁琐和痛苦的事情,Flux解决了这一问题。
特性
下面是Flux提供的所有的特性:
- 容易配置和部署拓扑(包括Storm和Trident)
- 支持变更已存在的拓扑
- 通过YAML文件来定义Spouts和Bolts,甚至可以支持Storm的其他组件,如storm-kafka/storm-hdfs/storm-hbase等
- 容易支持多语言协议组件
- 方便在不同环境中切换
使用
想要用Flux最简单的方法就是添加Maven依赖,然后打包成一个胖jar文件。依赖配置如下:
<code class="language-xml hljs has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"><!-- include Flux and user dependencies in the shaded jar --></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependencies</span>></span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"><!-- Flux include --></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.storm<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>flux-core<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>${storm.version}<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependency</span>></span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"><!-- add user dependencies here... --></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">dependencies</span>></span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"><!-- create a fat jar that includes all dependencies --></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">build</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">plugins</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">plugin</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span>org.apache.maven.plugins<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">groupId</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span>maven-shade-plugin<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">artifactId</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span>1.4<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">version</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">createDependencyReducedPom</span>></span>true<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">createDependencyReducedPom</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">executions</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">execution</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">phase</span>></span>package<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">phase</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goals</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goal</span>></span>shade<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goal</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">goals</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformers</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformer </span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">implementation</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"</span>/></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformer </span> <span class="hljs-attribute" style="box-sizing: border-box; color: rgb(102, 0, 102);">implementation</span>=<span class="hljs-value" style="box-sizing: border-box; color: rgb(0, 136, 0);">"org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"><<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">mainClass</span>></span>org.apache.storm.flux.Flux<span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">mainClass</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformer</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">transformers</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">configuration</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">execution</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">executions</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">plugin</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">plugins</span>></span> <span class="hljs-tag" style="color: rgb(0, 102, 102); box-sizing: border-box;"></<span class="hljs-title" style="box-sizing: border-box; color: rgb(0, 0, 136);">build</span>></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li></ul>
接下来是YAML文件的定义,一个拓扑的定义需要包括以下的部分:
- 拓扑名
- 拓扑组件的列表
- spouts、bolts、stream,或者是一个可以提供
org.apache.storm.generated.StormTopology
实例的JVM类。
下面是YAML文件实例:
<code class="language-yaml hljs applescript has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-property" style="box-sizing: border-box;">name</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"yaml-topology"</span> config: topology.workers: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># spout definitions</span> spouts: - <span class="hljs-property" style="box-sizing: border-box;">id</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spout-1"</span> className: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"org.apache.storm.testing.TestWordSpout"</span> parallelism: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># bolt definitions</span> bolts: - <span class="hljs-property" style="box-sizing: border-box;">id</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bolt-1"</span> className: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"org.apache.storm.testing.TestWordCounter"</span> parallelism: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> - <span class="hljs-property" style="box-sizing: border-box;">id</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bolt-2"</span> className: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"org.apache.storm.flux.wrappers.bolts.LogInfoBolt"</span> parallelism: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#stream definitions</span> streams: - <span class="hljs-property" style="box-sizing: border-box;">name</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spout-1 --> bolt-1"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># name isn't used (placeholder for logging, UI, etc.)</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"spout-1"</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bolt-1"</span> grouping: type: FIELDS args: [<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word"</span>] - <span class="hljs-property" style="box-sizing: border-box;">name</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bolt-1 --> bolt2"</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bolt-1"</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"bolt-2"</span> grouping: type: SHUFFLE </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li></ul>
在有了jar文件和YAML文件后就可以通过以下的命令运行Flux拓扑了,其中myTopology-0.1.0-SNAPSHOT.jar
是打包后的jar文件,org.apache.storm.flux.Flux
是Flux的入口类,--local
表示是在本地运行拓扑,my_config.yaml
使YAML配置文件。
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">storm jar myTopology-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span>-SNAPSHOT<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.jar</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.flux</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Flux</span> --local my_config<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.yaml</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
其他特性详解
不同环境切换
在不同的环境中运行拓扑需要不一样的配置,如开发环境和生产环境,这些环境中切换一般不会改变拓扑的结构,只是要修改主机、端口号和并行度等。如果用两份不一样的YAML文件来进行会产生不必要的重复,Flux可以通过.properites文件来加载不同的环境变量。只需要添加--filter
参数即可:
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">torm jar myTopology-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span>-SNAPSHOT<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.jar</span> org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.storm</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.flux</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Flux</span> --local my_config<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.yaml</span> --filter dev<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.properties</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
以YAML文件中的Kafka主机为例,YAML文件修改如下:
<code class="language-yaml hljs haml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">-<span class="ruby" style="box-sizing: border-box;"> <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">id:</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"zkHosts"</span> </span> className: "org.apache.storm.kafka.ZkHosts" constructorArgs: -<span class="ruby" style="box-sizing: border-box;"> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"${kafka.zookeeper.hosts}"</span></span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
而dev.properties问价如下:
<code class="hljs http has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-attribute" style="box-sizing: border-box;">kafka.zookeeper.hosts</span>: <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">localhost:2181</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
注:YAML文件中也可以解析系统环境变量${ENV-VARIABLE}
多语言协议的支持
多语言特性的支持比较简单,只需要修改YAML文件中构造参数,如下面是一个由Python写成的bolts:
<code class="language-yaml hljs haml has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">bolts: -<span class="ruby" style="box-sizing: border-box;"> <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">id:</span> <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"splitsentence"</span> </span> className: "org.apache.storm.flux.bolts.GenericShellBolt" constructorArgs: # command line -<span class="ruby" style="box-sizing: border-box;"> [<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"python"</span>, <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"splitsentence.py"</span>] </span> # output fields -<span class="ruby" style="box-sizing: border-box;"> [<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"word"</span>] </span> parallelism: 1</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li></ul>
展望
Flux虽然可以加方便拓扑的修改与部署,但这仍然不支持动态的修改拓扑结构,在修改拓扑时仍要中断并重启。不过现在在开发中的几个特性有望改善这个情况。