上一节(spark-core_28:Executor初始化过程env.blockManager.initialize(conf.getAppId)- NettyBlockTransferService.init()源码分析)
分析了
a、NettyBlockRpcServer是用于打开的上传注册在BlockManager中的Block块
b,TansportConf:会通过它成员:ConfigProvider子类关联SparkConf,并按SparkTransportConf.fromSparkConf(conf,"shuffle", numCores)设置nettyserver和client的线程池的大小,并按fromSparkConf第二个参数设置key的变量值
3,TransportContext:包含创建{TransportServer:nettyServer},{TransportClientFactory用来创建TransportClient}的上下文,并使用{TransportChannelHandler}设置Netty Channel管道。实例化TransportContext将给成员赋值 conf:TransportConf它可以通过成员子类ConfigProvider和sparkConf关联、rpcHandler:NettyBlockRpcServer、closeIdleConnections:false、同时给出站的编码器、入站解码器,赋具体实例
4,构造了TransportClientFactory:会将TransportContext、TransportConf,按IOMode的枚举类型,默认就是NIO, 得到NioSocketChannel.class基于IOMode枚举创建Netty的EventLoopGroup线程组、创建一个池化的ByteBuf分配器PooledByteBufAllocator给它的在成员变量
override def init(blockDataManager: BlockDataManager): Unit = {
/** conf.getAppId: app-20180508234845-0000
* serializer: JavaSerializer()
* blockDataManager : BlockManager实例
* NettyBlockRpcServer作用: 为每个请求打开或上传注册在BlockManager中的任意Block块,每一次Chunk的传输相当于一次shuffle
*/
val rpcHandler= new NettyBlockRpcServer(conf.getAppId, serializer, blockDataManager)
var serverBootstrap:Option[TransportServerBootstrap] = None
var clientBootstrap:Option[TransportClientBootstrap] = None
if (authEnabled) {//默认是false,不开启认证
serverBootstrap = Some(new SaslServerBootstrap(transportConf, securityManager))
clientBootstrap = Some(new SaslClientBootstrap(transportConf, conf.getAppId, securityManager,
securityManager.isSaslEncryptionEnabled()))
}
transportContext = new TransportContext(transportConf, rpcHandler)
/** 没有开启ssl所以clientBootstrap是空的
*
* TransportClientFactory:这个工厂实例通过使用createClient创建{TransportClient}。
* 这个工厂实例维护一个到其他主机的连接池,并应为相同的远程主机返回相同的TransportClient。 它还为所有TransportClient共享单个线程池。
*
* 在返回新客户端之前初始化运行给定TransportClientBootstraps的ClientFactory。Bootstraps会被同步执行,并且必须运行成功才能创建Client
* 给这个实例TransportClientFactory:赋成员
* context:TransportContext,
* conf:TransportConf会通过它成员:ConfigProvider子类关联SparkConf
* 还有初始化netty的NioSocketChannel.class、NioEventLoopGroup线程组、ByteBuf分配器PooledByteBufAllocator
*/
clientFactory = transportContext.createClientFactory(clientBootstrap.toSeq.asJava)
//创建一个nettySever,包括编解码,还有入站事件都加到TransportServer这个nettySever中
server =createServer(serverBootstrap.toList)
appId =conf.getAppId //值类似app-20180404172558-0000
logInfo("Servercreated on " + server.getPort)
1,先看一下NettyBlockTransferService.createServer()
/** Creates andbinds the TransportServer, possibly trying multiple ports.
* 创建并绑定TransportServer,可能尝试多个端口。*/
private def createServer(bootstraps: List[TransportServerBootstrap]):TransportServer = {
def startService(port: Int):(TransportServer, Int) = {
//实例化TransportContext将给成员赋值conf:TransportConf它可以通过成员子类ConfigProvider和sparkConf关联 、rpcHandler:NettyBlockRpcServer、closeIdleConnections:false、同时给出站的编码器、入站解码器,赋具体实例
val server= transportContext.createServer(port, bootstraps.asJava)
(server, server.getPort)
}
val portToTry= conf.getInt("spark.blockManager.port", 0)
//CoarseGrainedExecutorBackend的所有日志在%SPARK_HOME%/work下面
//18/05/16 16:33:47 INFO util.Utils:Successfully started service'org.apache.spark.network.netty.NettyBlockTransferService' on port 57010
//分配端口同时启动NettyServer
Utils.startServiceOnPort(portToTry, startService, conf, getClass.getName)._1
}
2,TransportContext.createServer(),先实例化TransportServer,它就是NettyServer
/** Create aserver which will attempt to bind to a specific port. */
public TransportServer createServer(int port, List<TransportServerBootstrap>bootstraps) {
/**
* context:就是当前TransportContext,可以创建{TransportServer},{TransportClientFactory}的上下文,并使用{TransportChannelHandler}设置Netty Channel管道
* hostToBind:null
* port: 是Utils.startServiceOnPort方法随机分配的
* rpcHandler:NettyBlockRpcServer作用: 为每个请求打开或上传注册在BlockManager中的任意Block块,每一次Chunk的传输相当于一次shuffle
* bootstraps:是一个空的ArrayList
*/
return new TransportServer(this, null, port, rpcHandler, bootstraps);
}
3,给TransportServer赋成员变量的同时,在当前节点上绑定端口
/**
* Creates a TransportServer that bindsto the given host and the given port, or to any available
* if 0. If you don't want to bind to anyspecial host, set "hostToBind" to null.
* 创建一个传输服务,绑定指定主机和端口,如果端口是0,则返回任何值,如果不想绑定主机,可以设置hostToBind为null
* context:就是当前TransportContext,可以创建{TransportServer},{TransportClientFactory}的上下文,并使用{TransportChannelHandler}设置Netty Channel管道
* hostToBind:null
* port: 是Utils.startServiceOnPort方法随机分配的
* rpcHandler:NettyBlockRpcServer作用: 为每个请求打开或上传注册在BlockManager中的任意Block块,每一次Chunk的传输相当于一次shuffle
* bootstraps:因为没有认证是一个空的List
* */
public TransportServer(
TransportContext context,
String hostToBind,
int portToBind,
RpcHandler appRpcHandler, //NettyBlockRpcServer
List<TransportServerBootstrap> bootstraps) {
this.context = context;
//conf: TransportConf会根据它的成员ConfigProvider子类关联到SparkConf
this.conf = context.getConf();
this.appRpcHandler= appRpcHandler;
this.bootstraps= Lists.newArrayList(Preconditions.checkNotNull(bootstraps));
try {
init(hostToBind, portToBind);
} catch(RuntimeException e) {
JavaUtils.closeQuietly(this);
throw e;
}
}
4,nettyserver的引导服务,可以查看netty官网
//hostToBind: null, portToBind:是Utils.startServiceOnPort方法随机分配的
private void init(String hostToBind, int portToBind){
//默认得到的NIO枚举值
IOMode ioMode = IOMode.valueOf(conf.ioMode());
//conf.serverThreads():CoarseGrainedExecutorBackend的core数相同
//该方法创建一个Netty的NioEventLoopGroup线程组
EventLoopGroup bossGroup =
NettyUtils.createEventLoop(ioMode, conf.serverThreads(), "shuffle-server");
EventLoopGroup workerGroup = bossGroup;
/**
* conf.preferDirectBufs(): 找key是:spark.shuffle.io.preferDirectBufs,查看SparkConf没有这个key则返回true
* conf.clientThreads()的值是1
*NettyUtils.createPooledByteBufAllocator():创建一个池化的ByteBuf分配器PooledByteBufAllocator
*/
PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(
conf.preferDirectBufs(), true /* allowCache */, conf.serverThreads());
//ServerBootstrap用于NIO服务端辅助启动类,说是降低服务端的开发复杂度
bootstrap = new ServerBootstrap()
.group(bossGroup, workerGroup)
.channel(NettyUtils.getServerChannelClass(ioMode))//NioServerSocketChannel.class
.option(ChannelOption.ALLOCATOR, allocator)
.childOption(ChannelOption.ALLOCATOR, allocator);
if (conf.backLog() > 0) {
//tcp通信队列的大小
bootstrap.option(ChannelOption.SO_BACKLOG, conf.backLog());
}
if (conf.receiveBuf() > 0) {
bootstrap.childOption(ChannelOption.SO_RCVBUF, conf.receiveBuf());
}
if (conf.sendBuf() > 0) {
bootstrap.childOption(ChannelOption.SO_SNDBUF, conf.sendBuf());
}
bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
//ChannelInitializer.initChannel()会在Channel注册后被调用,该方法结束后该ChannelInitializer拦截器会在ChannelPipeline移除
@Override
protected void initChannel(SocketChannelch) throws Exception {
//NettyBlockRpcServer作用: 为每个请求打开或上传注册在BlockManager中的任意Block块,每一次Chunk的传输相当于一次shuffle
RpcHandler rpcHandler = appRpcHandler;
for (TransportServerBootstrap bootstrap : bootstraps) {
rpcHandler =bootstrap.doBootstrap(ch, rpcHandler);
}
//context:TransportContext, 将SocketChannel和NettyBlockRpcServer放进去,给ChannelPipeline设置ChannelHander拦截器
//返回TransportChannelHandler是SimpleChannelInboundHandler子类,用于处理RequestMessages和ResponseMessages(这两个都是入站要处理的对象)的服务器端和客户端处理程序
context.initializePipeline(ch, rpcHandler);
}
});
//也就是说每个Worker对应的CoarseGrainedExecutorBackend的创建Executor时都会有一个Netty的server
//由于hostToBind是Null所以就是当前worker节点,用portToBind端口
InetSocketAddress address = hostToBind == null ?
new InetSocketAddress(portToBind):new InetSocketAddress(hostToBind, portToBind);
channelFuture = bootstrap.bind(address);
channelFuture.syncUninterruptibly();
port =((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
logger.debug("Shuffle server started on port:" + port);
}
5,TransportContext.initializePipeline(ch, rpcHandler)初始化ChannelPipeline的ChannelHandler拦截器
/**
* Initializes a client or server NettyChannel Pipeline which encodes/decodes messages and
* has a {@link org.apache.spark.network.server.TransportChannelHandler}to handle request or
* response messages.
*
* @param channel The channel to initialize.
* @param channelRpcHandler The RPC handler to use for the channel.
*
* @return Returns the createdTransportChannelHandler, which includes a TransportClient that can
* be used to communicate on thischannel. The TransportClient is directly associated with a
* ChannelHandler to ensure all users ofthe same channel get the same TransportClient object.
*
* 初始化客户端或服务器Netty Channel Pipeline里面编码/解码消息,并返回{TransportChannelHandler:它是一个SimpleChannelInboundHandler子类}来处理请求或响应消息。
*
* initializePipeline()返回创建的TransportChannelHandler,它包含可用于在此通道上通信的TransportClient。
* TransportClient直接与ChannelHandler相关联,以确保同一通道的所有用户都获得相同的TransportClient对象。
*
*==》TransportClient的作用:客户端获取预先协商的流的连续块。此API旨在允许大量数据的高效传输,将其分解为大小从几百KB到几MB的块。
* 该TransportClient类用于向服务器发送请求,而{@link TransportResponseHandler}负责处理来自服务器的响应,以响应[[TransportClient]]发出的请求。
*参数:
* channel:SocketChannel,通过的pipeline得到ChannelPipeline
* channelRpcHandler:NettyBlockRpcServer作用: 为每个请求打开或上传注册在BlockManager中的任意Block块,每一次Chunk的传输相当于一次shuffle
*/
public TransportChannelHandler initializePipeline(
SocketChannel channel,
RpcHandler channelRpcHandler) {
try {
//TransportChannelHandler是SimpleChannelInboundHandler子类,用于处理RequestMessages和ResponseMessages(这两个都是入站要处理的对象)的服务器端和客户端处理程序
TransportChannelHandler channelHandler =createChannelHandler(channel, channelRpcHandler);
channel.pipeline()
.addLast("encoder", encoder)
//spark自定义了tcp半包解码器
.addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
.addLast("decoder", decoder)
.addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs()/ 1000))
// NOTE: Chunks are currently guaranteed to be returnedin the order of request, but this
// would require more logic toguarantee if this were not part of the same event loop.
.addLast("handler", channelHandler);
return channelHandler;
} catch(RuntimeException e) {
logger.error("Error while initializing Nettypipeline", e);
throw e;
}
}
===>创建TransportContext.createChannelHandler()用于处理RequestMessages和ResponseMessages(这两个都是入站要处理的对象)的服务器端和客户端处理程序
/**
* Creates the server- and client-sidehandler which is used to handle both RequestMessages and ResponseMessages. Thechannel is expected to have been successfully created, though certain properties(such as the remoteAddress()) may not be available yet.
* 创建用于处理RequestMessages和ResponseMessages的服务器端和客户端处理程序。 预计该channel已成功创建,但某些属性(如remoteAddress())可能还不可用。
* 参数:
* channel:SocketChannel,通过的pipeline得到ChannelPipeline
* channelRpcHandler:NettyBlockRpcServer作用: 为每个请求打开或上传注册在BlockManager中的任意Block块,每一次Chunk的传输相当于一次shuffle
*/
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandlerrpcHandler) {
//TransportResponseHandler负责处理来自服务器的响应。以响应[[TransportClient]]发出的请求
TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
//该TransportClient类用于向服务器发送请求,TransportResponseHandler负责响应它的内容
TransportClient client = new TransportClient(channel, responseHandler);
//TransportRequestHandler:处理来自客户端的请求并将块数据写回的处理程序。
TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
rpcHandler);
//{TransportChannelHandler:它是一个SimpleChannelInboundHandler子类}来处理请求或响应消息
return new TransportChannelHandler(client, responseHandler, requestHandler,
conf.connectionTimeoutMs(), closeIdleConnections);
}
===》返回TransportServer.init(),通过serverBootStrap.bind()将nettyServer的端口绑定起来
//hostToBind: null, portToBind:是Utils.startServiceOnPort方法随机分配的
private void init(String hostToBind, int portToBind){
。。。。。
bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
//ChannelInitializer.initChannel()会在Channel注册后被调用,该方法结束后该ChannelInitializer拦截器会在ChannelPipeline移除
@Override
protected void initChannel(SocketChannelch) throws Exception {
//NettyBlockRpcServer作用: 为每个请求打开或上传注册在BlockManager中的任意Block块,每一次Chunk的传输相当于一次shuffle
RpcHandler rpcHandler = appRpcHandler;
for (TransportServerBootstrap bootstrap : bootstraps) {
rpcHandler =bootstrap.doBootstrap(ch, rpcHandler);
}
//context:TransportContext, 将SocketChannel和NettyBlockRpcServer放进去,给ChannelPipeline设置ChannelHander拦截器
//返回TransportChannelHandler是SimpleChannelInboundHandler子类,用于处理RequestMessages和ResponseMessages(这两个都是入站要处理的对象)的服务器端和客户端处理程序
context.initializePipeline(ch, rpcHandler);
}
});
//也就是说每个Worker对应的CoarseGrainedExecutorBackend的创建Executor时都会有一个Netty的server由于hostToBind是Null所以就是当前worker节点,用portToBind端口
InetSocketAddress address = hostToBind == null ?
new InetSocketAddress(portToBind):new InetSocketAddress(hostToBind, portToBind);
channelFuture = bootstrap.bind(address);
channelFuture.syncUninterruptibly();
port =((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
logger.debug("Shuffle server started on port:" + port);
}
==》到此NettyServer启动成功,同时整个NettyBlockTransferService.init()方法体也执行完成