Twitter Storm Distributed RPC

The idea behind distributed RPC (DRPC) is to parallelize the computation of really intense functions on the fly using Storm. The Storm topology takes in as input a stream of function arguments, and it emits an output stream of the results for each of those function calls.

DRPC is not so much a feature of Storm as it is a pattern expressed from Storm's primitives of streams, spouts, bolts, and topologies. DRPC could have been packaged as a separate library from Storm, but it's so useful that it's bundled with Storm.

DRPC概念早已经存在，不是Storm提出的新feature，DRPC主要用来解决intense functions，个人理解就是一个简单的输入会触发大量的后续操作（蝴蝶效应），如果将这样的计算放在单个节点肯定要耗很长时间，Storm DRPC利用我们自己定义的topology和给定的function arguments在Storm上计算出最终结果并返回给client。

High level overview

Distributed RPC is coordinated by a "DRPC server" (Storm comes packaged with an implementation of this).The DRPC server coordinates receiving an RPC request, sending the request to the Storm topology, receiving the results from the Storm topology, and sending the results back to the waiting client. From a client's perspective, a distributed RPC call looks just like a regular RPC call. For example, here's how a client would compute the results for the "reach" function with the argument "http://twitter.com":

DRPCClient client = new DRPCClient("drpc-host", 3772);
String result = client.execute("reach", "http://twitter.com");

The distributed RPC workflow looks like this:

A client sends the DRPC server the name of the function to execute and the arguments to that function. The topology implementing that function uses a DRPCSpout to receive a function invocation stream from the DRPC server. Each function invocation is tagged with a unique id by the DRPC server. The topology then computes the result and at the end of the topology a bolt called ReturnResults connects to the DRPC server and gives it the result for the function invocation id. The DRPC server then uses the id to match up that result with which client is waiting, unblocks the waiting client, and sends it the result.

从上面的图可以很好的知道DRPC的调用过程，其中起到关键作用的DRPC server，它主要负责接受client的请求、将请求扔到我们定义的topology上执行、从topology接收最终的计算结果、将计算结果返回给client。对于我们定义个topology是没有spout的，只会有bolts tree，至于spout，storm会将我们client提供的function name和arguments造一个DRPCSpout出来，这个DRPCSpout会发送streams给下游的bolts。

LinearDRPCTopologyBuilder

Storm comes with a topology builder called LinearDRPCTopologyBuilder that automates almost all the steps involved for doing DRPC. These include:

1. Setting up the spout
2. Returning the results to the DRPC server
3. Providing functionality to bolts for doing finite aggregations over groups of tuples

Let's look at a simple example. Here's the implementation of a DRPC topology that returns its input argument with a "!" appended:

public static class ExclaimBolt implements IBasicBolt {
    public void prepare(Map conf, TopologyContext context) {
    }

    public void execute(Tuple tuple, BasicOutputCollector collector) {
        String input = tuple.getString(1);
        collector.emit(new Values(tuple.getValue(0), input + "!"));
    }

    public void cleanup() {
    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("id", "result"));
    }

}

public static void main(String[] args) throws Exception {
    LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder("exclamation");
    builder.addBolt(new ExclaimBolt(), 3);
    // ...
}

As you can see, there's very little to it. When creating the LinearDRPCTopologyBuilder, you tell it the name of the DRPC function for the topology. A single DRPC server can coordinate many functions, and the function name distinguishes the functions from one another. The first bolt you declare will take in as input 2-tuples, where the first field is the request id and the second field is the arguments for that request.LinearDRPCTopologyBuilder expects the last bolt to emit an output stream containing 2-tuples of the form [id, result]. Finally, all intermediate tuples must contain the request id as the first field.

In this example, ExclaimBolt simply appends a "!" to the second field of the tuple. LinearDRPCTopologyBuilder handles the rest of the coordination of connecting to the DRPC server and sending results back.

LinearDRPCTopologyBuilder是storm提供的一个创建线性topology的builder，我们为它设定的第一个bolt的输入tuple包含两个fields，一个是request id，另外一个就是function arguments；而最后一个bolt的emit则是request id和result，这里需要注意的所有bolt的emit的第一个field都是request id并且全部相等。LinearDRPCTopologyBuilder会负责帮你连接DRPC server并将最后的结果发送给它。

Local mode DRPC

DRPC can be run in local mode. Here's how to run the above example in local mode:

LocalDRPC drpc = new LocalDRPC();
LocalCluster cluster = new LocalCluster();

cluster.submitTopology("drpc-demo", conf, builder.createLocalTopology(drpc)); // builder: LinearDRPCTopologyBuilder

System.out.println("Results for 'hello':" + drpc.execute("exclamation", "hello"));

cluster.shutdown();
drpc.shutdown();

First you create a LocalDRPC object. This object simulates a DRPC server in process, just like how LocalCluster simulates a Storm cluster in process. Then you create the LocalCluster to run the topology in local mode. LinearDRPCTopologyBuilder has separate methods for creating local topologies and remote topologies. In local mode the LocalDRPC object does not bind to any ports so the topology needs to know about the object to communicate with it. This is why createLocalTopology takes in the LocalDRPC object as input.

After launching the topology, you can do DRPC invocations using the execute method on LocalDRPC.

local模式：在单个进程内模拟的DRPC，主要用于开发阶段。

Remote mode DRPC

Using DRPC on an actual cluster is also straightforward. There's three steps:

1. Launch DRPC server(s)
2. Configure the locations of the DRPC servers
3. Submit DRPC topologies to Storm cluster

Launching a DRPC server can be done with the storm script and is just like launching Nimbus or the UI:

bin/storm drpc

Next, you need to configure your Storm cluster to know the locations of the DRPC server(s). This is how DRPCSpout knows from where to read function invocations. This can be done through the storm.yaml file or the topology configurations. Configuring this through the storm.yamllooks something like this:

drpc.servers:
  - "drpc1.foo.com"
  - "drpc2.foo.com"

Finally, you launch DRPC topologies using StormSubmitter just like you launch any other topology. To run the above example in remote mode, you do something like this:

StormSubmitter.submitTopology("exclamation-drpc", conf, builder.createRemoteTopology());

createRemoteTopology is used to create topologies suitable for Storm clusters.

Remote模式：需要启动DRPC server，在storm.yaml中配置drpc servers的位置，最后通过StormSubmitter提交topology。

A more complex example

The exclamation DRPC example was a toy example for illustrating the concepts of DRPC. Let's look at a more complex example which really needs the parallelism a Storm cluster provides for computing the DRPC function. The example we'll look at is computing the reach of a URL on Twitter.

The reach of a URL is the number of unique people exposed to a URL on Twitter. To compute reach, you need to:

1. Get all the people who tweeted the URL
2. Get all the followers of all those people
3. Unique the set of followers
4. Count the unique set of followers

A single reach computation can involve thousands of database calls and tens of millions of follower records during the computation. It's a really, really intense computation. As you're about to see, implementing this function on top of Storm is dead simple. On a single machine, reach can take minutes to compute; on a Storm cluster, you can compute reach for even the hardest URLs in a couple seconds.

A sample reach topology is defined in storm-starter here. Here's how you define the reach topology:

LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder("reach");
builder.addBolt(new GetTweeters(), 3);
builder.addBolt(new GetFollowers(), 12)
        .shuffleGrouping();
builder.addBolt(new PartialUniquer(), 6)
        .fieldsGrouping(new Fields("id", "follower"));
builder.addBolt(new CountAggregator(), 2)
        .fieldsGrouping(new Fields("id"));

The topology executes as four steps:

1. GetTweeters gets the users who tweeted the URL. It transforms an input stream of [id, url] into an output stream of [id, tweeter]. Each url tuple will map to many tweeter tuples.
2. GetFollowers gets the followers for the tweeters. It transforms an input stream of [id, tweeter] into an output stream of [id, follower]. Across all the tasks, there may of course be duplication of follower tuples when someone follows multiple people who tweeted the same URL.
3. PartialUniquer groups the followers stream by the follower id. This has the effect of the same follower going to the same task. So each task of PartialUniquer will receive mutually independent sets of followers. Once PartialUniquer receives all the follower tuples directed at it for the request id, it emits the unique count of its subset of followers.
4. Finally, CountAggregator receives the partial counts from each of the PartialUniquer tasks and sums them up to complete the reach computation.

Let's take a look at the PartialUniquer bolt:

public static class PartialUniquer implements IRichBolt, FinishedCallback {
    OutputCollector _collector;
    Map<Object, Set<String>> _sets = new HashMap<Object, Set<String>>();
    
    public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
        _collector = collector;
    }

    public void execute(Tuple tuple) {
        Object id = tuple.getValue(0);
        Set<String> curr = _sets.get(id);
        if(curr==null) {
            curr = new HashSet<String>();
            _sets.put(id, curr);
        }
        curr.add(tuple.getString(1));
        _collector.ack(tuple);
    }

    public void cleanup() {
    }

    public void finishedId(Object id) {
        Set<String> curr = _sets.remove(id);
        int count;
        if(curr!=null) {
            count = curr.size();
        } else {
            count = 0;
        }
        _collector.emit(new Values(id, count));
    }

    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("id", "partial-count"));
    }
}

When PartialUniquer receives a follower tuple in the execute method, it adds it to the set for the request id in an internal HashMap.

PartialUniquer also implements the FinishedCallback interface, which tells the LinearDRPCTopologyBuilder that it wants to be notified when it has received all of the tuples directed towards it for any given request id. This callback is the finishedId method. In the callback,PartialUniquer emits a single tuple containing the unique count for its subset of follower ids.

Under the hood, CoordinatedBolt is used to detect when a given bolt has received all of the tuples for any given request id. CoordinatedBoltmakes use of direct streams to manage this coordination.

The rest of the topology should be self-explanatory. As you can see, every single step of the reach computation is done in parallel, and defining the DRPC topology was extremely simple.

这里提供一个复杂些的例子，具体代码看最后部分。

Non-linear DRPC topologies

LinearDRPCTopologyBuilder only handles "linear" DRPC topologies, where the computation is expressed as a sequence of steps (like reach). It's not hard to imagine functions that would require a more complicated topology with branching and merging of the bolts. For now, to do this you'll need to drop down into using CoordinatedBolt directly. Be sure to talk about your use case for non-linear DRPC topologies on the mailing list to inform the construction of more general abstractions for DRPC topologies.

How LinearDRPCTopologyBuilder works

• DRPCSpout emits [args, return-info]. return-info is the host and port of the DRPC server as well as the id generated by the DRPC server
• constructs a topology comprising of:
  • DRPCSpout
  • PrepareRequest (generates a request id and creates a stream for the return info and a stream for the args)
  • CoordinatedBolt wrappers and direct groupings
  • JoinResult (joins the result with the return info)
  • ReturnResult (connects to the DRPC server and returns the result)
• LinearDRPCTopologyBuilder is a good example of a higher level abstraction built on top of Storm's primitives

原文地址：https://github.com/nathanmarz/storm/wiki/Distributed-RPC

附示例源代码：

import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.LocalDRPC;
import backtype.storm.StormSubmitter;
import backtype.storm.drpc.CoordinatedBolt.FinishedCallback;
import backtype.storm.drpc.LinearDRPCTopologyBuilder;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.topology.BasicOutputCollector;
import backtype.storm.topology.IBasicBolt;
import backtype.storm.topology.IRichBolt;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;

/**
 * This is a good example of doing complex Distributed RPC on top of Storm. This 
 * program creates a topology that can compute the reach for any URL on Twitter
 * in realtime by parallelizing the whole computation. 
 * 
 * Reach is the number of unique people exposed to a URL on Twitter. To compute reach,
 * you have to get all the people who tweeted the URL, get all the followers of all those people,
 * unique that set of followers, and then count the unique set. It's an intense computation 
 * that can involve thousands of database calls and tens of millions of follower records.
 * 
 * This Storm topology does every piece of that computation in parallel, turning what would be a 
 * computation that takes minutes on a single machine into one that takes just a couple seconds.
 * 
 * For the purposes of demonstration, this topology replaces the use of actual DBs with
 * in-memory hashmaps. 
 * 
 * See https://github.com/nathanmarz/storm/wiki/Distributed-RPC for more information on Distributed RPC.
 */
public class ReachTopology {
    public static Map<String, List<String>> TWEETERS_DB = new HashMap<String, List<String>>() {{
       put("foo.com/blog/1", Arrays.asList("sally", "bob", "tim", "george", "nathan")); 
       put("engineering.twitter.com/blog/5", Arrays.asList("adam", "david", "sally", "nathan")); 
       put("tech.backtype.com/blog/123", Arrays.asList("tim", "mike", "john")); 
    }};
    
    public static Map<String, List<String>> FOLLOWERS_DB = new HashMap<String, List<String>>() {{
        put("sally", Arrays.asList("bob", "tim", "alice", "adam", "jim", "chris", "jai"));
        put("bob", Arrays.asList("sally", "nathan", "jim", "mary", "david", "vivian"));
        put("tim", Arrays.asList("alex"));
        put("nathan", Arrays.asList("sally", "bob", "adam", "harry", "chris", "vivian", "emily", "jordan"));
        put("adam", Arrays.asList("david", "carissa"));
        put("mike", Arrays.asList("john", "bob"));
        put("john", Arrays.asList("alice", "nathan", "jim", "mike", "bob"));
    }};
    
    public static class GetTweeters implements IBasicBolt {
        @Override
        public void prepare(Map conf, TopologyContext context) {
        }

        @Override
        public void execute(Tuple tuple, BasicOutputCollector collector) {
            Object id = tuple.getValue(0);
            String url = tuple.getString(1);
            List<String> tweeters = TWEETERS_DB.get(url);
            if(tweeters!=null) {
                for(String tweeter: tweeters) {
                    collector.emit(new Values(id, tweeter));
                }
            }
        }

        @Override
        public void cleanup() {
        }

        @Override
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("id", "tweeter"));
        }        
    }
    
    public static class GetFollowers implements IBasicBolt {
        @Override
        public void prepare(Map conf, TopologyContext context) {
        }

        @Override
        public void execute(Tuple tuple, BasicOutputCollector collector) {
            Object id = tuple.getValue(0);
            String tweeter = tuple.getString(1);
            List<String> followers = FOLLOWERS_DB.get(tweeter);
            if(followers!=null) {
                for(String follower: followers) {
                    collector.emit(new Values(id, follower));
                }
            }
        }

        @Override
        public void cleanup() {
        }

        @Override
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("id", "follower"));
        }
    }
    
    public static class PartialUniquer implements IRichBolt, FinishedCallback {
        OutputCollector _collector;
        Map<Object, Set<String>> _sets = new HashMap<Object, Set<String>>();
        
        @Override
        public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
            _collector = collector;
        }

        @Override
        public void execute(Tuple tuple) {
            Object id = tuple.getValue(0);
            Set<String> curr = _sets.get(id);
            if(curr==null) {
                curr = new HashSet<String>();
                _sets.put(id, curr);
            }
            curr.add(tuple.getString(1));
            _collector.ack(tuple);
        }

        @Override
        public void cleanup() {
        }

        @Override
        public void finishedId(Object id) {
            Set<String> curr = _sets.remove(id);
            int count;
            if(curr!=null) {
                count = curr.size();
            } else {
                count = 0;
            }
            _collector.emit(new Values(id, count));
        }

        @Override
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("id", "partial-count"));
        }
    }
    
    public static class CountAggregator implements IRichBolt, FinishedCallback {
        Map<Object, Integer> _counts = new HashMap<Object, Integer>();
        OutputCollector _collector;
        
        @Override
        public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
            _collector = collector;
        }

        @Override
        public void execute(Tuple tuple) {
            Object id = tuple.getValue(0);
            int partial = tuple.getInteger(1);
            
            Integer curr = _counts.get(id);
            if(curr==null) curr = 0;
            _counts.put(id, curr + partial);
            _collector.ack(tuple);
        }

        @Override
        public void cleanup() {
        }

        @Override
        public void finishedId(Object id) {
            Integer reach = _counts.get(id);
            if(reach==null) reach = 0;
            _collector.emit(new Values(id, reach));
        }

        @Override
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("id", "reach"));
        }
        
    }
    
    public static LinearDRPCTopologyBuilder construct() {
        LinearDRPCTopologyBuilder builder = new LinearDRPCTopologyBuilder("reach");
        builder.addBolt(new GetTweeters(), 4);
        builder.addBolt(new GetFollowers(), 12)
                 .shuffleGrouping();
        builder.addBolt(new PartialUniquer(), 6)
                 .fieldsGrouping(new Fields("id", "follower"));
        builder.addBolt(new CountAggregator(), 3)
                 .fieldsGrouping(new Fields("id")); 
        return builder;
    }
    
    public static void main(String[] args) throws Exception {
        LinearDRPCTopologyBuilder builder = construct();
        
        
        Config conf = new Config();
        
        if(args==null || args.length==0) {
            conf.setMaxTaskParallelism(3);
            LocalDRPC drpc = new LocalDRPC();
            LocalCluster cluster = new LocalCluster();
            cluster.submitTopology("reach-drpc", conf, builder.createLocalTopology(drpc));
            
            String[] urlsToTry = new String[] { "foo.com/blog/1", "engineering.twitter.com/blog/5", "notaurl.com"};
            for(String url: urlsToTry) {
                System.out.println("Reach of " + url + ": " + drpc.execute("reach", url));
            }
            
            cluster.shutdown();
            drpc.shutdown();
        } else {
            conf.setNumWorkers(6);
            StormSubmitter.submitTopology(args[0], conf, builder.createRemoteTopology());
        }
    }
}