Storm 集群配置

最新推荐文章于 2024-04-18 17:05:17 发布

zhuhailong

最新推荐文章于 2024-04-18 17:05:17 发布

阅读量799

点赞数

分类专栏： Storm 文章标签： Storm 集群配置

本文链接：https://blog.csdn.net/u011317245/article/details/52523277

版权

Storm 专栏收录该内容

7 篇文章 0 订阅

订阅专栏

Storm集群配置分为以下步骤：

1）Set up a Zookeeper cluster

2）Install dependencies on Nimbus and worker machines

3）Download and extract a Storm release to Nimbus and worker machines

4）Fill in mandatory configurations into storm.yaml

5）Launch deamons under supervision using "storm" script and a supervisor of your choice

Set up a Zookeeper cluster：

Storm uses Zookeeper for coordinating(协调) the cluster . Zookeeper is not used for message passing . so the load Storm places on Zookeeper is quite low. Single node Zookeeper clusters should be sufficient(满足) for most cases . but if you want failover or are deploying large Storm cluster you may want larger Zookeeper clusters . Instructions for deploying Zookeeper are here.

Install dependencies on Nimbus and worker machines：

Java 7
Python 2.6.6

Download and extract a Storm release to Nimbus and worker machines：

Storm download address from here

Fill in mandatory configurations into storm.yaml：

1）storm.zookeeper.servers：Storm集群使用的Zookeeper地址

storm.zookeeper.servers:
  - "111.222.333.444"
  - "555.666.777.888"

2）storm.local.dir：NImbus和Supervisor进程用于存储少量状态,如jars、confs等的本地磁盘文件,需要提前创建好并赋予权限

storm.local.dir: "/mnt/storm"

3）nimbus.seeds：Storm集群Nimbus机器地址,各个Supervisor工作节点要知道那个机器是Nimbus,然后才能下载Topologies的 jars、confs等文件

nimbus.seeds: ["111.222.333.44"]

4）supervisor.slots.ports：用于定义每个Supervisor工作节点，可以运行worker的数量，配置好端口给即将运行的worker使用

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

Monitoring Health of Supervisors：

Storm提供了一种机制 admin可以在supervisor上配置一个脚本来定期的判断当前的node是否是健康状态.admin可以让supervisor判断一个node是否健康通过执行脚本位于storm.health.check.dir. 如果脚本检测到node时非健康状态,那么必须输出一个标准输出（with the string error）. supervisor将会定期执行脚本检查output, 如果检测出有 string error 信息, the supervisor will shut down any workers and exit .

if the supervisor is running with supervision "/bin/storm node-health-check" can be called to determine if the supervisor should be launched or if the node is unhealthy.