Hadoop RPC源码分析

Hadoop RPC ：与传统的RPC程序相似，分为Server端和Client端二个部分，下面将分别介绍其工作机制

1.Client端分析：
|-->Client(Class<? extends Writable> valueClass, Configuration conf, SocketFactory factory)
  |-->分别设置 maxIdleTime、tcpNoDelay、pingInterval值
  |-->this.conf = conf; ｜设置configuation
  |-->this.socketFactory = factory; ｜设定socketFactory仓库，用于socket连接
|-->call(Writable param, InetSocketAddress addr,Class<?> protocol, UserGroupInformation ticket) |与服务器建立连接
  |-->Call call = new Call(param);  ｜new一个Call对象，Call对象对于在RPC中作为基础的传输单元
  |-->Connection connection = getConnection(addr, protocol, ticket, call);
  |-->connection.sendParam(call);   ｜发送call数据
  |--> while (!call.done) 
	|--> call.wait();   |client端此处作同步处理，Server端实际为异步处理，在Response的setValue后被notify()唤醒
1.1 获取连接方法
  |-->getConnection |从连接池里面获取一个连接对象，没有则创建一个
	|-->ConnectionId remoteId = new ConnectionId(addr, protocol, ticket);  ｜创建连接id，避免重复分配connectionID
	|-->connection = new Connection(remoteId); ｜利用conncetionId创建Connectin连接
	|-->connections.put(remoteId, connection); ｜缓存进连接池当中,Hashtable予以保存
	|-->connection.setupIOstreams(); ｜在这此真正建立连接
1.1.1 connection.setupIOstreams()  ｜与服务器建立连接
|-->this.socket = socketFactory.createSocket();
|-->this.socket.setTcpNoDelay(tcpNoDelay);
|-->NetUtils.connect(this.socket, remoteId.getAddress(), 20000);  ｜用NIO的方式建立连接
	|-->SocketChannel ch = socket.getChannel();    ｜获取通道
	|-->socket.connect(endpoint, timeout);         ｜建立连接
|-->this.in = new DataInputStream(new BufferedInputStream         ｜分别设置输入输出流
            (new PingInputStream(NetUtils.getInputStream(socket))));        
|-->this.out = new DataOutputStream
            (new BufferedOutputStream(NetUtils.getOutputStream(socket)));
|-->writeHeader();  ｜写Connection的头信息
|-->touch();        ｜更新最后活动时间，同步等待responser进行处理，client模式为同步机制
|-->start();        ｜connection继承了Thread，实际上执行了run()方法
	|-->run()   
	   |-->while (waitForWork())  ｜等待服务器连接端响应，或者关闭连接
                 |-->receiveResponse();
		 |-->close()          ｜最后关闭
1.2 connection.sendParam(call);  ｜发送数据
|-->d = new DataOutputBuffer();    
|-->d.writeInt(call.id);          ｜首先写个call_id,用于标致唯一Call
|-->call.param.write(d);          ｜将Call中的Writable对象写入DataoutputStream中
|-->byte[] data = d.getData();     
|-->int dataLength = d.getLength(); ｜获取data的长度
|-->out.writeInt(dataLength);       ｜往通道中先写长度，再写数据
|-->out.write(data, 0, dataLength);
|-->out.flush();                    ｜flush，使写入生效
|-->IOUtils.closeStream(d);         ｜利用IOUtils关闭流

1.3 receiveResponse() ｜接收服务器端响应，处理请求。
|-->int id = in.readInt();
|-->Call call = calls.get(id);
|-->int state = in.readInt();  ｜如果为success状态，则读取in流
|-->Writable value = ReflectionUtils.newInstance(valueClass, conf);  |利用反射生成一个Writable对象
|-->value.readFields(in);
	|-->this.value = value;
	|-->callComplete();
	   |-->this.done = true;
	   |-->notify();  |此时用于唤醒在client端call()调用时的call.wait()
|-->call.setValue(value);
|-->calls.remove(id);



2.Server端分析：
2.1:namenode初始化Server端
|-->initialize()  |开启RPC的服务端
    |-->this.server = RPC.getServer(this, socAddr.getHostName(), socAddr.getPort(),
                                handlerCount, false, conf);
    |-->this.server.start();

2.1.1:RPC.getServer  |RPC的静态类Server是Server服务器类的子类
|-->new Server(instance, conf, bindAddress, port, numHandlers, verbose);
    |-->super(bindAddress, port, Invocation.class, numHandlers, conf, classNameBase(instance.getClass().getName()));
2.1.1.1:构建Server端类
|-->listener = new Listener();   ｜用于监听client端信息
|-->responder = new Responder(); ｜用于处理监听请求

2.2:this.server.start()  ｜启动server端服务，分别启动srepsonder,listener和handler类
|-->responder.start();   ｜注意启动顺序，确保监听器开始时即可处理响应请求
|-->listener.start();
|-->handlers = new Handler[handlerCount]; ｜handlerCount为handler线程个数 
|-->for (int i = 0; i < handlerCount; i++)
	|-->handlers[i] = new Handler(i);
	|-->handlers[i].start();

2.3:Listener监听器
2.3.1：构建Listener()
|-->address = new InetSocketAddress(bindAddress, port);
|-->acceptChannel = ServerSocketChannel.open();
|-->acceptChannel.configureBlocking(false);
|-->bind(acceptChannel.socket(), address, backlogLength);
|-->selector= Selector.open();
|-->acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
|-->this.setDaemon(true);   ｜设置为后台守护进程

注：上述构建过程实际是简单的NIO server端启动过程，当client端请求过多时会出现瓶颈，后续版本增加了reader
|-->readers = new Reader[readThreads];  
|-->readPool = Executors.newFixedThreadPool(readThreads);   |设定reader线程池大小
|-->for (int i = 0; i < readThreads; i++)           
	|-->Selector readSelector = Selector.open();  
	|-->Reader reader = new Reader(readSelector);  
	|-->readers[i] = reader;  
	|-->readPool.execute(reader);  

2.3.2:Listener线程run()执行
|-->while (running)   |无限轮循
|-->selector.select();
|-->terator<SelectionKey> iter = selector.selectedKeys().iterator();
|-->while (iter.hasNext())
	|-->key = iter.next();
        |-->iter.remove();
	|-->if (key.isAcceptable())
		|-->doAccept(key);  ｜往通道里面写数据
	|-->else if (key.isReadable())
		|-->doRead(key);    ｜从通道里面读取数据

2.3.3:doAccept(key) ｜接受client请求
|-->ServerSocketChannel server = (ServerSocketChannel) key.channel();
|-->for (int i=0; i<10; i++)  ｜开启最多10个线程进行处理
|-->SocketChannel channel = server.accept();
|-->channel.configureBlocking(false);
|-->SelectionKey readKey = channel.register(selector, SelectionKey.OP_READ); |此channel重新注册至selector中，进行读取操作
|-->c = new Connection(readKey, channel, System.currentTimeMillis());
|-->readKey.attach(c);  ｜将connection附加到key当中,在is_readable中继续处理

2.3.4:doRead(key)   ｜处理client请求
|-->Connection c = (Connection)key.attachment();    ｜在accept时作为attach封装了Connection信息
|-->count = c.readAndProcess();
	|-->dataLengthBuffer   |如果未满或已经为空，则读取了一个RPC，否则继续读
	|-->channelRead(channel, dataLengthBuffer);
	|-->int version = versionBuffer.get(0);  ｜读取版本信息
	|-->data = ByteBuffer.allocate(dataLength);  ｜获取rpc大小
	|-->if (headerRead)  ｜判断headerRead是否读取，默认都会先发送header文件
		|-->processData();
	|-->else |先处理header头信息
		|-->processHeader();
|-->closeConnection(c);

2.3.4.1 processHeader() 
|-->header.readFields(in);
|-->String protocolClassName = header.getProtocol();  ｜从头文件中获取protocl的处理class
|-->protocol = getProtocolClass(header.getProtocol(), conf);
|-->user = SecurityUtil.getSubject(header.getUgi());  ｜获取user信息，作为安全策略

2.3.4.2 processData()
|-->int id = dis.readInt();
|-->Writable param = ReflectionUtils.newInstance(paramClass, conf); |paramClass此时为namenode的class
|-->param.readFields(dis);
|-->Call call = new Call(id, param, this);   |在服务器端构建Call单元，并加入等待处理队列 
|-->callQueue.put(call);       ｜加入callQueue队列后，工作将交由handler进行处理



2.4:Handler处理器
2.4.1 构建Handler处理器 ,简单设置了Thread信息
	|-->this.setDaemon(true);
	|-->this.setName("IPC Server handler "+ instanceNumber + " on " + port);
2.4.2 run()线程执行
|-->while (running)
|-->final Call call = callQueue.take();
|-->CurCall.set(call); ｜确保当前处理的call唯一
|-->value = Subject.doAs(call.connection.user, new PrivilegedExceptionAction<Writable>() {
                 @Override
                 public Writable run() throws Exception 
                      return call(call.connection.protocol, call.param, call.timestamp);｝
|-->setupResponse(buf, call,  (error == null) ? Status.SUCCESS : Status.ERROR, 
                        value, errorClass, error);       ｜此时设置call的response信息
|-->responder.doRespond(call); ｜调用responder响应处理完成的call请求
			
2.4.3 执行call方法调用
call(call.connection.protocol, call.param, call.timestamp)
|-->Invocation call = (Invocation)param;  |此时param对象预先读取了method方法名及其参数
|-->Method method = protocol.getMethod(call.getMethodName(),
                                   call.getParameterClasses());
|-->method.setAccessible(true);
|-->Object value = method.invoke(instance, call.getParameters());  ｜利用反射获取方法调用的值
|-->return new ObjectWritable(method.getReturnType(), value);      ｜最后返回一个ObjectWritable的处理结果

2.4.4 responder.doRespond(call)  ｜属于responder的处理策略，提前分析
|-->call.connection.responseQueue.addLast(call);
|-->call.connection.responseQueue == 1 ｜call的responseQueue队列为1，则处理call单元
	|-->processResponse(call.connection.responseQueue, true);   ｜下节分析

2.5 Responser处理器

2.5.1 processResponse(responseQueue，inHandler)   |handler处理完call数据后，传递给Responser
|-->call = responseQueue.removeFirst();
|-->SocketChannel channel = call.connection.channel;
|-->int numBytes = channelWrite(channel, call.response);
|-->if (!call.response.hasRemaining()) |此时response中已经不存在call对象了
	|-->call.connection.decRpcCount();
|-->else 
	|-->call.connection.responseQueue.addFirst(call);
	|-->writeSelector.wakeup();  ｜激活响应通道
        |-->channel.register(writeSelector, SelectionKey.OP_WRITE, call);

2.5.2 run() ｜Thread执行方法
|-->waitPending();       ｜等待wake up唤醒
|-->writeSelector.select(PURGE_INTERVAL);  ｜设置超时
|-->Iterator<SelectionKey> iter = writeSelector.selectedKeys().iterator();
|-->while (iter.hasNext())
	|-->SelectionKey key = iter.next();
	|-->iter.remove();
	|--> doAsyncWrite(key);

|-->calls = new ArrayList<Call>(writeSelector.keys().size());
|-->while (iter.hasNext())
	|-->Call call = (Call)key.attachment(); 
	|-->calls.add(call);
|--> for(Call call : calls) 
	|-->doPurge(call, now); ｜清除响应队列中长久未响应的call


2.5.3  doAsyncWrite(key)
|-->Call call = (Call)key.attachment();
|-->if (processResponse(call.connection.responseQueue, false))  ｜调用processResponse进行响应