组通信之jgroups篇----Building Blocks

Building blocks are layered on top of channels. Most of them do not even need a channel, all they need is a class that implements interface Transport (channels do). This enables them to work on any type of group transport that obeys this interface. Building blocks can be used instead of channels whenever a higher-level interface is required. Whereas channels are simple socket-like constructs, building blocks may offer a far more sophisticated interface. In some cases, building blocks offer access to the underlying channel, so that -- if the building block at hand does not offer a certain functionality -- the channel can be accessed directly. Building blocks are located in the org.jgroups.blocks package. Only the ones that are relevant for application programmers are discussed below.

 

 

Building blocks位于channel上层.他们中的大部分甚至不需要channel.他们都需要的是一个Transport接口的实现(channel也是Transport接口的实现).所以他们可以位于任何遵循此接口的组通信.channel是一个简单的类似套接字的结构,而Building blocks则提供了更多复杂的接口.有些Building blocks可以直接使用底层的channel.Building blocks在包org.jgroups.blocks中.下面讨论的都将是应用程序开发者将会使用到的.

 

4.1. PullPushAdapter
Note that this building block has been deprecated and should not be used anymore !

This class is a converter (or adapter, as used in [Gamma:1995] between the pull-style of actively receiving messages from the channel and the push-style where clients register a callback which is invoked whenever a message has been received. Clients of a channel do not have to allocate a separate thread for message reception.

A PullPushAdapter is always created on top of a class that implements interface Transport (e.g. a channel). Clients interested in being called when a message is received can register with the PullPushAdapter using method setListener(). They have to implement interface MessageListener, whose receive() method will be called when a message arrives. When a client is interested in getting view, suspicion messages and blocks, then it must additionally register as a MembershipListener using method setMembershipListener(). Whenever a view, suspicion or block is received, the corresponding method will be called.

Upon creation, an instance of PullPushAdapter creates a thread which constantly calls the receive() method of the underlying Transport instance, blocking until a message is available. When a message is received, if there is a registered message listener, its receive() method will be called.

As this class does not implement interface Transport, but merely uses it for receiving messages, an underlying object has to be used to send messages (e.g. the channel on top of which an object of this class resides). This is shown in Figure 4.1, “Class PullPushAdapter”.

As is shown, the thread constantly pulls messages from the channel and forwards them to the registered listeners. An application thus does not have to actively pull for messages, but the PullPushAdapter does this for it. Note however, that the application has to directly access the channel if it wants to send a message.

 

注意:此building block已经被移除并将不再使用.

该类是一个转换器或适配器,用于将从channel拿消息的pull模式转换为通过注册回调函数,当有消息接收到时被调用的push模式.这样,客户端就不需要专门线程去接收消息了

PullPushAdapter创建一般都需要Transport接口的实现类(如channel).客户端可以通过PullPushAdapter的setListener函数去注册.他们需要实现MessageListener接口,这样,当有消息到达时,其实现类的receive函数将被调用.当然,如果客户端对视图,怀疑消息,阻塞消息有兴趣,同样可以通过setMembershipListener函数注册MembershipListener实现类,当有相关消息接收到时,相应的方法就会被调用.

具体实现是,PullPushAdapter实例将创建一个线程专门调用底层Transport实例的receive函数,阻塞的接收消息.当有消息到达且注册有消息监听器,则注册的receive函数将被调用.

因为此类没有实现Transport接口,而仅仅是用于接收消息,所以,得用其底层对象去发送消息(如channel)(个人认为:该类已经实现了发送函数).

此类有专门的线程用于从channel中pull出消息并将传给已注册的监听器.所以应用程序才不用自己去pull消息.(后面这句应用程序需通过channel发送消息是正确的,但其意思是必须直接使用channel发送是不对的,如上所说,该类自己其实也有发送函数).

 

4.1.1. Example
This section shows sample code for using a PullPushAdapter. The example has been shortened for readability (error handling has been removed).

    public class PullPushTest implements MessageListener {
        Channel          channel;
        PullPushAdapter  adapter;
        byte[]           data="Hello world".getBytes();
        String           props; // fetch properties

        public void receive(Message msg) {
            System.out.println("Received msg: " + msg);
        }

        public void start() throws Exception {
            channel=new JChannel(props);
            channel.connect("PullPushTest");
            adapter=new PullPushAdapter(channel);
            adapter.setListener(this);

            for(int i=0; i < 10; i++) {
                System.out.println("Sending msg #" + i);
                channel.send(new Message(null, null, data));
                Thread.currentThread().sleep(1000);
            }
            adapter.stop();
            channel.close();
        }
        public static void main(String args[]) {
            try {
                new PullPushTest().start();
            }
            catch(Exception e) { /* error */ }
        }
    } 
First a channel is created and connected to. Then an instance of PullPushAdapter is created with the channel as argument. The constructor of PullPushAdapter starts its own thread which continually reads on the channel. Then the MessageListener is set, which causes all messages received on the channel to be sent to receive(). Then a number of messages are sent via the channel to the entire group. As group messages are also received by the sender, the receive() method will be called every time a message is received. Finally the PullPushAdapter is stopped and the channel closed. Note that explicitly stopping the PullPushAdapter is not actually necessary, a closing the channel would cause the PullPushAdapter to terminate anyway.Note that, compared to the pull-style example, push-style message reception is considerably easier (no separate thread management) and requires less code to program.

Note
The PullPushAdapter has been deprecated, and will be removed in 3.0. Use a Receiver implementation instead. The advantage of the Receiver-based (push) model is that we save 1 thread.

首先,channel被创建并连接.然后, PullPushAdapter对象被创建,以channel为参数. PullPushAdapter启动自己的线程去接收消息.然后设置MessageListener,它可以将接收到的消息转到监听器的receive函数.然后,一系列消息将经过channel发送到整个组.由于组内消息也会被发送端自己接收,所以,自己的receive接口也会被调用.最后, PullPushAdapter被停止并且关闭channel.

注意: PullPushAdapter已被移除,3.0版本将不再使用.而是使用Receiver类的实现代替.其好处是可以继续节省一个线程.(具体可以参考上一章

jgroups之JChannel的3.6.10. Using a Receiver to receive messages).

 

 

4.2. MessageDispatcher

Channels are simple patterns to asynchronously send a receive messages. However, a significant number of communication patterns in group communication require synchronous communication. For example, a sender would like to send a message to the group and wait for all responses. Or another application would like to send a message to the group and wait only until the majority of the receivers have sent a response, or until a timeout occurred.

MessageDispatcher offers a combination of the above pattern with other patterns. It provides synchronous (as well as asynchronous) message sending with request-response correlation, e.g. matching responses with the original request. It also offers push-style message reception (by internally using the PullPushAdapter).

An instance of MessageDispatcher is created with a channel as argument. It can now be used in both client and server role: a client sends requests and receives responses and a server receives requests and send responses. MessageDispatcher allows a application to be both at the same time. To be able to serve requests, the RequestHandler.handle() method has to be implemented:
        Object handle(Message msg);

The handle() method is called any time a request is received. It must return a return value (must be serializable, but can be null) or throw an exception. The return value will be returned to the sender (as a null response, see below). The exception will also be propagated to the requester.

The two methods to send requests are:
        public RspList castMessage(Vector dests, Message msg, int mode, long timeout);
        public Object sendMessage(Message msg, int mode, long timeout) throws TimeoutException;

The castMessage() method sends a message to all members defined in dests. If dests is null the message will be sent to all members of the current group. Note that a possible destination set in the message will be overridden. If a message is sent synchronously then the timeout argument defines the maximum amount of time in milliseconds to wait for the responses.

The mode parameter defines whether the message will be sent synchronously or asynchronously. The following values are valid (from org.jgroups.blocks.GroupRequest):

GET_FIRST

Returns the first response received.

GET_ALL

Waits for all responses (minus the ones from suspected members)

GET_MAJORITY

Waits for a majority of all responses (relative to the group size)

GET_ABS_MAJORITY

Waits for the majority (absolute, computed once)

GET_N

Wait for n responses (may block if n > group size)

GET_NONE

Wait for no responses, return immediately (non-blocking). This make the call asynchronous.

The sendMessage() method allows an application programmer to send a unicast message to a receiver and optionally receive the response. The destination of the message has to be non-null (valid address of a receiver). The mode argument is ignored (it is by default set to GroupRequest.GET_FIRST) unless it is set to GET_NONE in which case the request becomes asynchronous, ie. we will not wait for the response.

One advantage of using this building block is that failed members are removed from the set of expected responses. For example, when sending a message to 10 members and waiting for all responses, and 2 members crash before being able to send a response, the call will return with 8 valid responses and 2 marked as failed. The return value of castMessage() is a RspList which contains all responses (not all methods shown):

    public class RspList {

        public boolean isReceived(Address sender);

        public int     numSuspectedMembers();

        public Vector  getResults();

        public Vector  getSuspectedMembers();

        public boolean isSuspected(Address sender);

        public Object  get(Address sender);

        public int     size();

        public Object  elementAt(int i) throws ArrayIndexOutOfBoundsException;

    }  

Method isReceived() checks whether a response from sender has already been received. Note that this is only true as long as no response has yet been received, and the member has not been marked as failed. numSuspectedMembers() returns the number of members that failed (e.g. crashed) during the wait for responses. getResults() returns a list of return values. get() returns the return value for a specific member.

 

channel是简单的异步发送接收模式,但是,很大部分的组通信需要同步通信.比如,发送端可能需要发送一个消息到组并等待所有回应,或者发送端应用程序发送消息到组并等待主要接收端返回或直到超时产生.

MessageDispatcher提供了上面模式的实现,它能实现同步消息发送接收(也能实现异步模式),并能正确对应请求-回应关系.它同样提供了push模式的消息接收方法(通过使用PullPushAdapter).

MessageDispatcher实例的创建以channel为参数.它同时可以用于客户端和服务器端.客户端发送请求并接收响应,服务器端接收请求并发送响应.MessageDispatcher允许应用程序同时既是发送端也是服务端.我们必须实现RequestHandler.handle方法以处理请求.

handle方法在有请求被接收时调用,它需要返回一个值(需序列化,可以为空)或者抛出异常.该返回值将会返回给发送端,出错同样会被发送给请求者.

有两种方法发送请求:castMessage和sendMessage.

castMessage函数发送消息到定义的发送端的所有成员.如果发送端为空,则消息将发送到当前组所有成员.如果消息同步发送则超时参数指等待响应的毫秒数.模式参数不区分消息是同步还是异步发送,其值如下:

 

GET_FIRST  只要第一个成员响应即返回

GET_ALL  等待所有成员响应

GET_MAJORITY  等待主要成员响应(跟组大小有关).

GET_ABS_MAJORITY  等待主要成员响应(绝对成员,计算结果)

GET_N 等待N个成员响应(如果N>组大小数,则会阻塞).

GET_NONE 不等待任何成员,立即返回,异步调用.

sendMessage方法允许单播一个消息到接收端,并可选择是否等待回应.消息的发送端不能为空,模式参数忽略除非是GET_NONE,否则它表示不等待任何消息,为异步方式.

 

此building block有一个好处是失败成员将会从期望的响应者中去除.比如,发送消息给10个成员,有2个成员异常不能发送响应,则返回8个有效的响应,那2个标志为失败. castMessage的返回值为RspList,它包含所有响应成员.

isReceived方法用于检查发送端发出的响应消息是否被接收到,如果没有响应消息被接收到且没有成员标记为失败,则返回值为真. numSuspectedMembers返回失败的成员个数. getResults返回一个值的列表,get函数返回某个特定成员的返回值.

 

 

 

4.2.1. Example

This section describes an example of how to use a MessageDispatcher.

    public class MessageDispatcherTest implements RequestHandler {

        Channel            channel;

        MessageDispatcher  disp;

        RspList            rsp_list;

        String             props; // to be set by application programmer

        public void start() throws Exception {

            channel=new JChannel(props);

            disp=new MessageDispatcher(channel, null, null, this);

            channel.connect("MessageDispatcherTestGroup");

            for(int i=0; i < 10; i++) {

                Util.sleep(100);

                System.out.println("Casting message #" + i);

                rsp_list=disp.castMessage(null,

                    new Message(null, null, new String("Number #" + i)),

                    GroupRequest.GET_ALL, 0);

                System.out.println("Responses:/n" +rsp_list);

            }

            channel.close();

            disp.stop();

        }

        public Object handle(Message msg) {

            System.out.println("handle(): " + msg);

            return new String("Success !");

        }

 

        public static void main(String[] args) {

            try {

                new MessageDispatcherTest().start();

            }

            catch(Exception e) {

                System.err.println(e);

            }

        }

    }    

The example starts with the creation of a channel. Next, an instance of MessageDispatcher is created on top of the channel. Then the channel is connected. The MessageDispatcher will from now on send requests, receive matching responses (client role) and receive requests and send responses (server role).

We then send 10 messages to the group and wait for all responses. The timeout argument is 0, which causes the call to block until all responses have been received.

The handle() method simply prints out a message and returns a string.

Finally both the MessageDispatcher and channel are closed.

 

下面的例子说明了如何使用MessageDispatcher.

首先,创建channel.然后, MessageDispatcher实例基于channel被创建.再连接channel. MessageDispatcher现在将可用来发送请求,接收相匹配的回应(客户端)或者接收请求发送响应(服务器角色).

然后我们发送10个消息到组并等待所有回应.超时为0表示阻塞知道所有响应都被接收.

handle函数仅仅打印出接收到的消息.

然后,关闭MessageDispatcher和channel.

 

 

4.3. RpcDispatcher

This class is derived from MessageDispatcher. It allows a programmer to invoke remote methods in all (or single) group members and optionally wait for the return value(s). An application will typically create a channel and layer the RpcDispatcher building block on top of it, which allows it to dispatch remote methods (client role) and at the same time be called by other members (server role).

Compared to MessageDispatcher, no handle() method needs to be implemented. Instead the methods to be called can be placed directly in the class using regular method definitions (see example below). The invoke remote method calls (unicast and multicast) the following methods are used (not all methods shown):

    public RspList callRemoteMethods(Vector dests, String method_name, int mode, long timeout);

    public RspList callRemoteMethods(Vector dests, String method_name, Object arg1, int mode, long timeout);

    public Object callRemoteMethod(Address dest, String method_name,int mode, long timeout);

    public Object callRemoteMethod(Address dest, String method_name, Object arg1, int mode, long timeout);

The family of callRemoteMethods() is invoked with a list of receiver addresses. If null, the method will be invoked in all group members (including the sender). Each call takes the name of the method to be invoked and the mode and timeout parameters, which are the same as for MessageDispatcher. Additionally, each method takes zero or more parameters: there are callRemoteMethods() methods with up to 3 arguments. As shown in the example above, the first 2 methods take zero and one parameters respectively.

The family of callRemoteMethod() methods takes almost the same parameters, except that there is only one destination address instead of a list. If the dest argument is null, the call will fail.

If a sender needs to use more than 3 arguments, it can use the generic versions of callRemoteMethod() and callRemoteMethods() which use a MethodCall instance rather than explicit arguments.

Java's Reflection API is used to find the correct method in the receiver according to the method name and number and types of supplied arguments. There is a runtime exception if a method cannot be resolved.

(* Update: these methods are deprecated; must use MethodCall argument now *)

 

此类继承自MessageDispatcher.它允许所有成员(或一个)调用远程方法,并可选择是否等待返回值.应用程序一般需创建channel然后基于此实现RpcDispatcher building block.它允许发送远程消息(客户端角色)或被其他成员调用某函数(服务器角色).

与MessageDispatcher不同,这里没有handle方法需要实现.要被调用的方法可以直接在此类的一些函数中定义,有如下函数可以实现远程函数调用.

callRemoteMethods方法会被列表中的接收地址响应,如果该列表为空,被调用方法将被所有成员响应(包含发送端).每个方法包含方法名,模式和超时参数,这个跟MessageDispatcher相同.

callRemoteMethod方法和callRemoteMethods方法参数类似(此处介绍函数参数缺少了Class[] types),只不过此函数的目的地是一个地址而不是一个列表,且不能为空.

如果参数太多,可以使用MethodCall参数.

JAVA映射API用于找到接收端的正确方法,通过参数名和参数类型.如果没有相应函数则有运行时异常产生.

(现在这些方法也被移除,要使用MethodCall参数)

 

4.3.1. Example

The code below shows an example:

    public class RpcDispatcherTest {

        Channel            channel;

        RpcDispatcher      disp;

        RspList            rsp_list;

        String             props; // set by application

        public int print(int number) throws Exception {

            return number * 2;

        }

        public void start() throws Exception {

            channel=new JChannel(props);

            disp=new RpcDispatcher(channel, null, null, this);

            channel.connect("RpcDispatcherTestGroup");

 

            for(int i=0; i < 10; i++) {

                Util.sleep(100);

                rsp_list=disp.callRemoteMethods(null, "print",

                     new Integer(i), GroupRequest.GET_ALL, 0);

                System.out.println("Responses: " +rsp_list);

            }

            channel.close();

            disp.stop();

         }

        public static void main(String[] args) {

            try {

                new RpcDispatcherTest().start();

            }

            catch(Exception e) {

                System.err.println(e);

            }

        }

    }

Class RpcDispatcher defines method print() which will be called subsequently. The entry point start() method creates a channel and an RpcDispatcher which is layered on top. Method callRemoteMethods() then invokes the remote print() method in all group members (also in the caller). When all responses have been received, the call returns and the responses are printed.

As can be seen, the RpcDispatcher building block reduces the amount of code that needs to be written to implement RPC-based group communication applications by providing a higher abstraction level between the application and the primitive channels.

 

RpcDispatcher类定义了print函数,此函数将被调用.入口函数start创建了channel和RpcDispatcher对象,RpcDispatcher位于channel之上.callRemoteMethods方法将调用远程的print函数,当所有响应消息均被接收到时,此方法返回.

可以看到,RpcDispatcher building block提供了更高的抽象层,减少了代码实现基于组通信的RPC调用.

 

4.4. ReplicatedHashMap

This class was written as a demo of how state can be shared between nodes of a cluster. It has never been heavily tested and is therefore not meant to be used in production, and unsupported.

A ReplicatedHashMap uses a concurrent hashmap internally and allows to create several instances of hashmaps in different processes. All of these instances have exactly the same state at all times. When creating such an instance, a group name determines which group of replicated hashmaps will be joined. The new instance will then query the state from existing members and update itself before starting to service requests. If there are no existing members, it will simply start with an empty state.

Modifications such as put(), clear() or remove() will be propagated in orderly fashion to all replicas. Read-only requests such as get() will only be sent to the local copy.

Since both keys and values of a hashtable will be sent across the network, both of them have to be serializable. This allows for example to register remote RMI objects with any local instance of a hashtable, which can subsequently be looked up by another process which can then invoke remote methods (remote RMI objects are serializable). Thus, a distributed naming and registration service can be built in just a couple of lines.

A ReplicatedHashMap allows to register for notifications, e.g. when a new item is set, or an existing one removed. All registered listeners will notified when such an event occurs. Notification is always local; for example in the case of removing an element, first the element is removed in all replicas, which then notify their listener(s) of the removal (after the fact).

ReplicatedHashMap allow members in a group to share common state across process and machine boundaries.

 

此类用于在组内节点共享状态信息.该类没有经过严格测试,不推荐在产品上使用,缺乏支持.

ReplicatedHashMap内部使用了并发hashmap,允许在不同进程创建多个hashmap实例.这些实例一直保持相同的状态信息.但创建某个实例时,组名决定它加入共享哪个hashmap,该实例在启动服务前将从已存在成员获取状态信息.如果此时还没有成员,则它以空状态启动.

put,clear,remove等消息方法将被顺序的传递到所有成员中,get等只读方法将拷贝给本地使用.

由于hashtable的键值需要网络传输,则他们需被序列化.它允许以本地实例的hashtable去注册远程的RMI对象,随后被另一进程得到并调用远程方法(具体意思可以理解为注册远程对象,其和本地对象结构是一致的,然后其他成员会被响应而调用具体方法).于是,一个分布式的命名和注册服务只需几行即可实现了.

ReplicatedHashMap还可以注册事件通知事件,当有成员加入或退出时,所有注册的监听器都将获得通知,通知一般是本地事件.比如,当有成员退组时,首先,该成员会在所有成员中被移除出组,然后才通知本地的监听器该移除动作已发生.

ReplicatedHashMap的作用就是允许组内成员共享通用数据.

(此类集合了以前的ReplicatedHashtable异步共享数据和DistributedHashtable同步共享数据的功能).

 

4.5. NotificationBus
This class provides notification sending and handling capability. Also, it allows an application programmer to maintain a local cache which is replicated by all instances. NotificationBus also sits on top of a channel, however it creates its channel itself, so the application programmers do not have to provide their own channel. Notification consumers can subscribe to receive notifications by calling setConsumer() and implementing interface NotificationBus.Consumer:

    public interface Consumer {
        void          handleNotification(Serializable n);
        Serializable  getCache();
        void          memberJoined(Address mbr);
        void          memberLeft(Address mbr);
    }
Method handleNotification() is called whenever a notification is received from the channel. A notification is any object that is serializable. Method getCache() is called when someone wants to retrieve our state; the state can be returned as a serializable object. The memberJoined() and memberLeft() callbacks are invoked whenever a member joins or leaves (or crashes).

The most important methods of NotificationBus are:

    public class NotificationBus {
         public void setConsumer(Consumer c);
         public void start() throws Exception;
         public void stop();
         public void sendNotification(Serializable n);
         public Serializable getCacheFromCoordinator(long timeout, int max_tries);
         public Serializable getCacheFromMember(Address mbr, long timeout, int max_tries);
    } 
Method setConsumer() allows a consumer to register itself for notifications.

The start() and stop() methods start and stop the NotificationBus.

Method sendNotification() sends the serializable object given as argument to all members of the group, invoking their handleNotification() methods on reception.

Methods getCacheFromCoordinator() and getCacheFromMember() provide functionality to fetch the group state from the coordinator (first member in membership list) or any other member (if its address is known). They take as arguments a timeout and a maximum number of unsuccessful attempts until they return null. Typically one of these methods would be called just after creating a new NotificationBus to acquire the group state. Note that if these methods are used, then the consumers must implement Consumer.getCache(), otherwise the two methods above would always return null.

 

此类提供了一个发送和接收通知的功能.它也能允许一个应用程序编程者维护本地cache数据到组内所有成员.NotificationBus同样基于channel之上,但它自己内部创建channel,不需要我们自己提供channel作参数,通过setConsumer函数设置Notification consumer,我们需要实现其接口,这样,我们就可以在有通知事件到来时被响应了.

当从channel接收到数据时,handleNotification函数被调用,通知对象是被序列化的.当有成员需要获取状态时,getCache函数被调用.当有成员加入或离开时,memberJoined和memberLeft方法会被调用.

NotificationBus的主要方法如下:

setConsumer方法允许注册consumer接收通知事件.

start和stop方法启动和停止NotificationBus.

sendNotification方法以一个序列化对象为参数发送给所有成员,接收端handleNotification方法会被响应.

getCacheFromCoordinator和getCacheFromMember方法提供了从主(第一个成员)或其他成员获取组状态功能.它提供了超时和尝试次数参数.它主要用于在创建NotificationBus后获取组状态,如果使用该方法,我们必须实现Consumer.getCache接口.

 

总结:
PullPushAdapter主要提供了一个专门的接收线程,从而将pull模式转换为push模式.但其接收过程本质上还是阻塞式的,消息到达时先放入队列,然后接收线程从队列拿数据,再通知已注册的MessageListener.

Receiver本文未细说,但这是取代PullPushAdapter的接口.它通过setReceiver函数在Channel层注册自己,这样,我们就不需要维护队列,有消息到来时,Receiver接口的实现类相关函数会被直接调用,所以,它同时也节省了PullPushAdapter的接收线程.

MessageDispatcher是用于实现同步消息发送,它需要等待此消息是否被各成员接收.它通过setUpHandler函数在Channel层注册自己.它能区分别的成员发送的消息和它需要的回应消息等.
RpcDispatcher主要用于远程RPC方法调用,继承自MessageDispatcher类。同样是需要等待结果。同样在Channel层注册自己。
ReplicatedHashMap主要维护了一个组的共享的通用数据.
NotificationBus则提供了更抽象的一层,它已经不需要用户关心channel等底层.它能维护组的一个共享cache(可以是组状态,组共享数据),同时,它也能完成组的消息和事件通知机制.