hadoop中的RPC应用示例

hadoop中的RPC意义

      前面有讲到RPC的基础内容，还给出了一个小的示例来说明，这篇文章讲讲hadoop中的RPC是怎样使用的。
      RPC在Hadoop中是基础组件，提供分布式环境下的对象调用功能。可以把它理解成胶水，由于它的存在，分布式系统才互相进行通信。当然这个胶水可以被其它类型的胶水替换，也可以自己去做这个胶水。
      胶水很重要，不能缺少，当然如果只是把hadoop作为一个单机来运行，那要不要无所谓了。说了很多废话，总结一下就是说，RPC对分布式的hadoop而言很重要。

my code

首先说明一下，以下程序跑起来需要哪些jar包：

jdk不用说了，还有
hadoop-core
hadoop-client
hadoop-common
commons-logging
commons-collections
commons-configuration
commons-lang

1.server

package hadooprpc;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.ipc.RPC;
import org.apache.hadoop.ipc.RPC.Server;

public class MyServer {

    final static int port = 8888;

    final static String address = "127.0.0.1";

    public static void main(String args[]) {

        Proxy proxy = new Proxy();
        try {
            Server server = RPC.getServer(proxy, address, port, new Configuration());
            server.start();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

2.client

package hadooprpc;

import java.io.IOException;
import java.net.InetSocketAddress;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.ipc.RPC;

public class Client {

    final static int port = 8888;

    final static String address = "127.0.0.1";

    public static void main(String[] args) {

        InetSocketAddress inetSocketAddress = new InetSocketAddress(address, port);

        try {

            MyJob job = (MyJob) RPC.waitForProxy(MyJob.class, MyJob.versionId, inetSocketAddress, new Configuration());
            String re = job.getInfo("zj");
            System.out.println(re);

        } catch (IOException e) {

            e.printStackTrace();
        }
    }
}

3.utils

package hadooprpc;

import org.apache.hadoop.ipc.VersionedProtocol;

public interface MyJob extends VersionedProtocol{

    long versionId = 123456;
    public String getInfo(String a);
}


package hadooprpc;

import java.io.IOException;

public class Proxy implements MyJob {

    @Override
    public String getInfo(String a) {

        return "你好： " + a;
    }

    @Override
    public long getProtocolVersion(String arg0, long arg1) throws IOException {
        return MyJob.versionId;
    }

}

可以看到整个框架和之前在讲述RPC时差不多，区别在于，hadoop将socket服务封装了，因此我们使用的时候就简单很多。

hadoop的RPC server实现

RPC.getServer()的调用指向

return new Server(instance, conf, bindAddress, port, numHandlers, verbose, secretManager);

server中的参数有

    private final boolean authorize;
    private boolean isSecurityEnabled;
    private ExceptionsHandler exceptionsHandler;
    public static final ByteBuffer HEADER = ByteBuffer.wrap("hrpc".getBytes());
    public static final byte CURRENT_VERSION = 4;
    private static final int IPC_SERVER_HANDLER_QUEUE_SIZE_DEFAULT = 100;
    private static final String IPC_SERVER_HANDLER_QUEUE_SIZE_KEY = "ipc.server.handler.queue.size";
    static int INITIAL_RESP_BUF_SIZE = 10240;
    static final String IPC_SERVER_RPC_MAX_RESPONSE_SIZE_KEY = "ipc.server.max.response.size";
    static final int IPC_SERVER_RPC_MAX_RESPONSE_SIZE_DEFAULT = 1048576;
    public static final Log LOG = LogFactory.getLog(Server.class);
    private static final Log AUDITLOG = LogFactory.getLog("SecurityLogger." + Server.class.getName());
    private static final String AUTH_FAILED_FOR = "Auth failed for ";
    private static final String AUTH_SUCCESSFULL_FOR = "Auth successfull for ";
    private static final ThreadLocal<Server> SERVER = new ThreadLocal();
    private static final Map<String, Class<?>> PROTOCOL_CACHE = new ConcurrentHashMap();
    private static final ThreadLocal<Call> CurCall = new ThreadLocal();
    private String bindAddress;
    private int port;
    private int handlerCount;
    private int readThreads;
    private Class<? extends Writable> paramClass;
    private int maxIdleTime;
    private int thresholdIdleConnections;
    int maxConnectionsToNuke;
    protected RpcInstrumentation rpcMetrics;
    private Configuration conf;
    private SecretManager<TokenIdentifier> secretManager;
    private int maxQueueSize;
    private final int maxRespSize;
    private int socketSendBufferSize;
    private final boolean tcpNoDelay;
    private volatile boolean running;
    private BlockingQueue<Call> callQueue;
    private List<Connection> connectionList;
    private Listener listener;
    private Responder responder;
    private int numConnections;
    private Handler[] handlers;
    private static int NIO_BUFFER_LIMIT = 8192;

   //它们的初始化情况为，使用构造器初始化
    this.exceptionsHandler = new ExceptionsHandler();
    this.running = true;
    this.connectionList = Collections.synchronizedList(new LinkedList());
    this.listener = null;
    this.responder = null;
    this.numConnections = 0;
    this.handlers = null;
    this.bindAddress = bindAddress;
    this.conf = conf;
    this.port = port;
    this.paramClass = paramClass;
    this.handlerCount = handlerCount;
    this.socketSendBufferSize = 0;
    this.maxQueueSize = handlerCount * conf.getInt("ipc.server.handler.queue.size", 100);
    this.maxRespSize = conf.getInt("ipc.server.max.response.size", 1048576);
    this.readThreads = conf.getInt("ipc.server.read.threadpool.size", 1);
    this.callQueue = new LinkedBlockingQueue(this.maxQueueSize);
    this.maxIdleTime = 2 * conf.getInt("ipc.client.connection.maxidletime", 1000);
    this.maxConnectionsToNuke = conf.getInt("ipc.client.kill.max", 10);
    this.thresholdIdleConnections = conf.getInt("ipc.client.idlethreshold", 4000);
    this.secretManager = secretManager;
    this.authorize = conf.getBoolean("hadoop.security.authorization", false);
    this.isSecurityEnabled = UserGroupInformation.isSecurityEnabled();
    this.listener = new Listener(this);
    this.port = this.listener.getAddress().getPort();
    this.rpcMetrics = RpcInstrumentation.create(serverName, this.port);
    this.tcpNoDelay = conf.getBoolean("ipc.server.tcpnodelay", false);
    this.responder = new Responder(this);
    if (this.isSecurityEnabled) {
        SaslRpcServer.init(conf);
    }

server.start();该指令的指向为

public synchronized void start() {
    this.responder.start();
    this.listener.start();
    this.handlers = new Handler[this.handlerCount];

    for (int i = 0; i < this.handlerCount; ++i) {
        this.handlers[i] = new Handler(this, i);
        this.handlers[i].start();
    }

}

那说明responder，listener，handler其实是线程服务。一个返回线程服务，一个监听器服务，多个处理器服务。

那么线程和线程之间的交互是怎么做到的呢？

private BlockingQueue<Call> callQueue;
private List<Connection> connectionList;

答案是通过这些队列。
————————————————————————————————
后面的代码是eclipse反编译得到的，后面的代码就不好获取。之后将会在hadoop源代码中去分析。

参考内容：

1.https://www.cnblogs.com/qq503665965/p/6708644.html
2.https://www.cnblogs.com/edisonchou/p/4285817.html