1.Storm 在taobao的使用情况:
We make statistics of logs and extract useful information from thestatistics in almost real-time with Storm. Logs are read from Kafka-likepersistent message queues into spouts, then processed and emitted over thetopologies to compute desired results, which are then stored into distributeddatabases to be used elsewhere. Input log count varies from 2 millions to 1.5billion every day, whose size is up to 2 terabytes among the projects. The mainchallenge here is not only real-time processing of big data set; storing andpersisting result is also a challenge and needs careful design andimplementation.
淘宝使用storm和消息队列结合,每天能够处理2百万到15亿条日志,日志量达到2TB的近实时处理。
2.使用场景
上周开始学习storm的使用,现在探索出来两种使用场景。
1, 通过配置drpc服务器,将storm的topology发布为drpc服务。客户端程序可以调用drpc服务将数据发送到storm集群中,并接收处理结果的反馈。这种方式需要drpc服务器进行转发,其中drpc服务器底层通过thrift实现。适合的业务场景主要是实时计算。并且扩展性良好,可以增加每个节点的工作worker数量来动态扩展。
2, 第二种场景是通过beanstalkd来实现信息的导入,将topology任务提交到storm集群后可以通过开发beanstalkd客户端来向集群中发送信息,这种方式客户端收不到结果反馈。这个场景适合纯粹的数据分析处理的业务场景。
3.Strom drpc服务配置:
端口可以不用配置,默认是:3772
Nimbus节点的配置:
storm.zookeeper.servers:
- "10.10.249.195"
- "10.10.249.196"
#
# nimbus.host: "nimbus"
## Locations of the drpc servers
drpc.servers:
- "10.10.249.197"
# - "server2"
Supervisor节点的配置:
########### These MUST be filled in for astorm configuration
storm.zookeeper.servers:
- "10.10.249.195"
- "10.10.249.196"
#
nimbus.host: "10.10.249.195"
#
## Locations of the drpc servers
drpc.servers:
- "10.10.249.197"
# - "server2"
supervisor.slots.ports:
-6700
-6701
- 6702
Drpc服务器节点配置
该节点只需配置zookeeper地址即可。默认开放的端口:3772
storm.zookeeper.servers:
-"10.10.249.195"
-"10.10.249.196"
启动drpc服务:./storm drpc