Hadoop RPC源码分析之Server

最新推荐文章于 2024-09-21 21:27:42 发布

置顶 xhjx2618

最新推荐文章于 2024-09-21 21:27:42 发布

阅读量473

点赞数

文章标签： hadoop

本文链接：https://blog.csdn.net/xhjx2618/article/details/8500567

版权

Server.Listener内部类

这个内部类监听服务器Socket,看是否有来自客户端的连接,读取通道中的数据(实际上是调用Connction的方法来读取数据的).内部使用了readThreads个Listener.Reader线程来读取所有的请求数据,每个Reader线程中有一个 Selector readSelector成员变量,在这个readSelector上面可以注册多个SocketChannel,并且增加SelectionKey.OP_READ事件,在reader线程里面的run方法里,不断地检测该reader的readSelector上注册的各个通道上是否有可读数据,如果有的话就调用doRead(key)方法读取数据.这样的做的好处就是只是使用了固定个(readThreads个)reader线程来读取请求,而不用像传统的socket通信那样,为每个请求创建一个线程来处理,明显降低系统开销.(使用java nio的原因?)

构造一个Listener:

public Listener() throws IOException {
			address = new InetSocketAddress(bindAddress, port);
			// Create a new server socket and set to non blocking mode
			acceptChannel = ServerSocketChannel.open();
			acceptChannel.configureBlocking(false);

			// Bind the server socket to the local host and port
			bind(acceptChannel.socket(), address, backlogLength);
			port = acceptChannel.socket().getLocalPort(); // Could be an
															// ephemeral port
			// create a selector;
			selector = Selector.open();
			
			//创建一个reader线程池
			readers = new Reader[readThreads];
			readPool = Executors.newFixedThreadPool(readThreads);
			for (int i = 0; i < readThreads; i++) {
				Selector readSelector = Selector.open();
				Reader reader = new Reader(readSelector);
				readers[i] = reader;
				readPool.execute(reader);
			}

			//将这个listener的acceptChannel注册到它的selector上,将且这个acceptChannel上注册OP_ACCEPT事件,在Listener的run()方法中调用selector.select()方法
			acceptChannel.register(selector, SelectionKey.OP_ACCEPT);
			this.setName("IPC Server listener on " + port);
			this.setDaemon(true);
		}

下面来看下Server端到底做了什么事情:

1. 在上面提到的Listener的构造方法中,首先创建的reader线程池用来读取来自客户端的请求数据(此时还没有将channel注册到reader上), 然后调用 acceptChannel.register(selector, SelectionKey.OP_ACCEPT) ,这一代码将其这个listener的acceptChannel(ServerSocketChannel类型)注册到它的selector上,并且注册了OP_ACCEPT事件,在Listener的run()方法中调用selector.select()方法,检测是否有新的连接到来,如果有的话,则调用doAccept方法接求连接.看一下run方法:

public void run() {
			LOG.info(getName() + ": starting");
			SERVER.set(Server.this);
			while (running) {
				SelectionKey key = null;
				try {
					selector.select();
					Iterator<SelectionKey> iter = selector.selectedKeys().iterator();
					while (iter.hasNext()) {
						key = iter.next();
						iter.remove();
						try {
							if (key.isValid()) {
								if (key.isAcceptable())  如果该key对应的通道已经准备好接收新的Socket连接
									doAccept(key);// 调用，接收与该key关联的通道上的连接
							}
						} catch (IOException e) {
						}
						key = null;
					}
				} catch (OutOfMemoryError e) {
					// we can run out of memory if we have too many threads
					// log the event and sleep for a minute and give
					// some thread(s) a chance to finish
					LOG.warn("Out of Memory in server select", e);
					closeCurrentConnection(key, e);
					cleanupConnections(true);
					try {
						Thread.sleep(60000);
					} catch (Exception ie) {
					}
				} catch (Exception e) {
					closeCurrentConnection(key, e);
				}
				cleanupConnections(false);
			}
			LOG.info("Stopping " + this.getName());

			synchronized (this) {
				try {
					acceptChannel.close();
					selector.close();
				} catch (IOException e) {
				}

				selector = null;
				acceptChannel = null;

				// clean up all connections
				while (!connectionList.isEmpty()) {
					closeConnection(connectionList.remove(0));
				}
			}
		}

2.在doAccept方法里面,获取客户端的socketChannel,通过轮转的方式在reader线程池中获取一个reader,将客户端的socketChannel注册到这个reader中,即调用 reader.registerChannel(channel),在这个方法内部,将这个channel注册到reader的readSelector上,并且注册了SelectionKey.OP_READ事件,至此,这个reader可以准备读取通道数据了.

//在doAccept方法中,为一个reader的selector注册相应的通道(SocketChannel类型),那么在reader的run方法中会调用doRead(key)方法
		void doAccept(SelectionKey key) throws IOException, OutOfMemoryError {
			Connection c = null;
			ServerSocketChannel server = (ServerSocketChannel) key.channel();//向上追溯可以发现,传给doAccept方法的key对应的通道是ServerSocketChannel类型的
			SocketChannel channel;
			while ((channel = server.accept()) != null) {  //channel是客户端的socket通道
				channel.configureBlocking(false);
				channel.socket().setTcpNoDelay(tcpNoDelay);
				Reader reader = getReader();//轮转的方式获得一个Reader线程
				try {
					reader.startAdd();
					SelectionKey readKey = reader.registerChannel(channel);  //这里将客户端的socket通道注册到reader线程的Selector对象上,并且为这个通道注册了OP_READ事件,即在reader线程中可以读取来自客户端的消息
					c = new Connection(readKey, channel,System.currentTimeMillis());//用这个channel构造一个Connection对象
					readKey.attach(c);
					
					synchronized (connectionList) {
						connectionList.add(numConnections, c);
						numConnections++;
					}
					if (LOG.isDebugEnabled())
						LOG.debug("Server connection from " + c.toString()
								+ "; # active connections: " + numConnections
								+ "; # queued calls: " + callQueue.size());
				} finally {
					reader.finishAdd();
				}

			}
		}

这里用到了一个Reader内部类,它是前面构造的线程池的运行类,主要用于读取socketChannel中的数据,它有一个成员:private Selector readSelector = null,在前面的的doAccept方法中为这个selector成员注册相应的socketChannel

3.reader线程启动了以后,在它的run方法中不断检测自身对应的通道上是否有新的数据可读,如果有的话则调用doRead(key)读取通道数据:

public void run() {
				LOG.info("Starting SocketReader");
				synchronized (this) {
					while (running) {
						SelectionKey key = null;
						try {
							readSelector.select();
							while (adding) {
								this.wait(1000);
							}

							Iterator<SelectionKey> iter = readSelector.selectedKeys().iterator();
							while (iter.hasNext()) {
								key = iter.next();
								iter.remove();
								if (key.isValid()) {
									if (key.isReadable()) { //
										doRead(key);
									}
								}
								key = null;
							}
						} catch (InterruptedException e) {
							if (running) { // unexpected -- log it
								LOG.info(getName() + " caught: "
										+ StringUtils.stringifyException(e));
							}
						} catch (IOException ex) {
							LOG.error("Error in Reader", ex);
						}
					}
				}
			}

4.在doRead方法中,参数key对应的通道类型是SocketChannel类型的(在doAccept中注册的),这里最终还是调用connection的readAndProcess()方法读取一个call加入的Server的callQueue中

//读取一个通道时的数据,传给这个方法的key对应的通道类型是SocketChannel类型的(在doAccept中注册的),这里最终还是调用connection的readAndProcess()方法读取一个call加入的Server的callQueue中
		void doRead(SelectionKey key) throws InterruptedException {
			int count = 0;
			Connection c = (Connection) key.attachment();
			if (c == null) {
				return;
			}
			c.setLastContact(System.currentTimeMillis());

			try {
				count = c.readAndProcess();//读取一个call加入的Server的callQueue中
			} catch (InterruptedException ieo) {
				LOG.info(getName()
						+ ": readAndProcess caught InterruptedException", ieo);
				throw ieo;
			} catch (Exception e) {
				LOG.info(getName() + ": readAndProcess threw exception " + e
						+ ". Count of bytes read: " + count, e);
				count = -1; // so that the (count < 0) block is executed
			}
			if (count < 0) {
				if (LOG.isDebugEnabled())
					LOG.debug(getName() + ": disconnecting client " + c
							+ ". Number of active connections: "
							+ numConnections);
				closeConnection(c);
				c = null;
			} else {
				c.setLastContact(System.currentTimeMillis());
			}
		}

5. 在Server.connection的 readAndProcess中,经过层层调用,最终会调用connection的private void processData(byte[] buf)方法,这个方法处理一个call,将这个call加入到Server的callQueue中,通过反序列化操作从网络字节流中冲重构调用参数数据对象，并构造Server.Call对象，同时加入callQueue队列，等待Server.Handler线程进行处理

private void processData(byte[] buf) throws IOException,InterruptedException {
			DataInputStream dis = new DataInputStream(new ByteArrayInputStream(buf));
			int id = dis.readInt(); // try to read an id
			if (LOG.isDebugEnabled())
				LOG.debug(" got #" + id);
			Writable param = ReflectionUtils.newInstance(paramClass, conf);// read param
			param.readFields(dis);
			Call call = new Call(id, param, this);
			callQueue.put(call); // queue the call; maybe blocked here
			incRpcCount(); // Increment the rpc count
		}

6. 在Handler的run方法中,从callQueue中取出一个call,调用call(Class<?> protocol, Writable param,long receiveTime) 方法对这个call进行响应处理,这个call方法由具体的Server实现类实现的;然后,调用Server的setupResponse(ByteArrayOutputStream response, Call call,Status status, Writable rv, String errorClass, String error)方法,将call调用的返回值写入到call对象的private ByteBuffer response 变量中,最后,调用responder.doRespond(call)将call对象加入到call对应的connection对象的responseQueue对象里面(call.connection.responseQueue),将call交给Responder处理,向客户端返回请求结果(processResponse方法将结果写回通道).

总结一下:

Listener : 负责所有的I/O事件,监听所有来自客户端的连接,然后将任务分发到各个Reader线程

Listener.Reader: 读取自身管理的通道上的所有channel上的请求

Connection: 负责到远程客户端的连接,实现了从对应channel中读取一个call将基加入到Server的callQueue队列的功能

Handler : 从callQueue队列中取出一个call对象,调用Server的call方法,响应call请求,将结果写入call对象的response字段

Responder : 向发送请求的客户端写回响应数据