Storm

Overview

Easily defining the level of parallelism for each bolt and spout across the cluster, so that you can scale your topology indefinitely.

Terms

  • Spout: input stream of Storm is handled by a component called Spout.
  • Bolt: Spout passes the data to the component called Bolt, which transforms the data in some way.
  • Topology: the arrangement of all the components (Spout and Bolt) and their connections is called Topology

Node

There are two type of nodes in Storm cluster:

  • master node: run a daemon called Nimbus, which is responsible for
    • distributing the code across the cluster
    • assigning the tasks to each node
    • monitoring for failure
  • worker node: run a daemon called Supervisor, which executes a portion of a topology. 

Underneath, Storm makes use of ZeroMQ.

Sample

Spout

public class WordReader extends BaseRichSpout {

    private SpoutOutputCollector collector;
    private FileReader fileReader;
    
    // the first method gets called by Storm framewokr
    public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
        // context: contains all Topology data
        // conf: is created in Topology definition
        this.fileReader = new FileReader(conf.get("file_name").toString())
        // collector: enable us to emit the data that will be processed by bolts
        this.collector = collector;
    }

    // declare the output fields
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("field_name"));
    }

    public void close() {
    }

    // emits values that will be processed by bolts
    public void nextTuple() {
        ...
        fileReader.read();
        ...
        // Fields is defined in the declareOutputFields .
        collector.emit(new Values("field_vale"));
        ...
    }

    public void ack(Object msgId) {
    }

    public void fail(Object msgId) {
    }
}

Bolt

public class WordNormalizer extends BaseBasicBolt {

    public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("field_name"));
    }

    public void cleanup() {
    }

    public void execute(Tuple input, BasicOutputCollector collector) {
        // use field name or index
        String str = input.getString("spout_output_field_name")
        // process str
        ...
        // emit the value or persist the message.
        collector.emit(new Values("value"));
        // acknolowedge the tuple or fail
        collector.ack(input);
    }
}

Main

public class TopologyMain {

    public static void main(String[] args) throws InterruptedException {         
        // Configuration
        Config conf = new Config();
        conf.setDebug(true);
        conf.put("wordsFile", args[0]);
        conf.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1);

        // Topology Definition
        TopologyBuilder builder = new TopologyBuilder();
        builder.setSpout("word-reader",new WordReader());
        builder.setBolt("word-normalizer", new WordNormalizer()).shuffleGrouping("word-reader");
        builder.setBolt("word-counter", new WordCounter(), 1).fieldsGrouping("word-normalizer", new Fields("word"));

        // Define the cluster
        LocalCluster cluster = new LocalCluster();

        // Run Topology
        cluster.submitTopology("Getting-Started-Toplogie", conf, builder.createTopology());
        Thread.sleep(1000);

        // Shutdown
        cluster.shutdown();
    }
}

 

转载于:https://my.oschina.net/u/3551123/blog/1492194

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值