RPC in oslo.messaging

Author: Lubin Liu

This post introduces the logic of rpc client in oslo.messaging, which is widely used in Openstack ecological system.

1. Overview of RPC in oslo.messaging

The following picture is quoted from openstack official website, and you could read this link to understand the components in this picture.

The basic idea of RPC in oslo.messaging
The full name of RPC is "remote procedure call", and it is used widely in distributed systems.The "invoker" is the client side, and a client usually call a method and the method is run on the "worker", i.e., the server side.
Two kind of RPC semantics are supported in oslo:
• Call: a blocking RPC request, and the client will wait for the response from the server side.
• Cast: a non-blocking RPC request, and the client won't wait for any response.
In oslo.messaing, the RPC is implemented with messaging middle-ware. For "call" semantic, two queues will be involved. The client and server are appointed to some queue, and the client side will put its request into this queue. The server receives the request and run corresponding logic and generates the response. The response is put into the reply queue. The client side pull the response and finish this round RPC.

For "cast" semantic, only one queue will be involved. Only the appointed queue, no reply queue.

With the help of messaging middle-ware, the client request and the server response is asynchronous and decoupling.

More details for call semantic
We take rabbitmq as an example of messaging middle-ware and introduce more details in a call semantic.
1. initialization: the rpc client and rpc server construct the transport object and listen on the same topic.
2. the 'msg publisher' in rpc client sends the request to the appointed topic by specify the target for this request. The request also specifies the name of the reply queue and the message id.

3. the 'msg consumer' in rpc server pulls the request and handle it.
4. the rpc server starts the direct publisher and send the response to the reply queue. The response is bound to the request message id.
5. the 'direct consumer' in rpc client pulls the response from the reply queue and give the response to proper call thread based on the message id.

2. Analyze the implementation on source code level
The following sections, I will analyze part of the source code. The code you can find here. It is very similar between RPC client and RPC server, so I take the client side as an example.

RPCClient

# constructor
def __init__(self, transport, target,
timeout=None, version_cap=None, serializer=None, retry=None):
# how to use it
class TestClient(object):
def __init__(self, transport):
target = messaging.Target(topic='test', version='2.0')
self._client = messaging.RPCClient(transport, target)
def test(self, ctxt, arg):
return self._client.call(ctxt, 'test', arg=arg)

The comments in the source code are very clear. RPCClient has two important methods. "call", which invokes a method and wait for a reply. "cast", which invokes a method and return immediately. The RPCClient is an abstraction, and the implementation is in "_CallContext". Let's take a look at the implementations of the "call" method and "cast" method.

def call(self, ctxt, method, **kwargs):
...
result = self.transport._send(self.target, msg_ctxt, msg,
wait_for_reply=True, timeout=timeout,
retry=self.retry)
...
return self.serializer.deserialize_entity(ctxt, result)
def cast(self, ctxt, method, **kwargs):
...
self.transport._send(self.target, ctxt, msg, retry=self.retry)
...