BlockTransferService
Spark是分布式部署的,每个Task最终都运行在不同的机器节点上。map任务的输出结果直接存储到map任务所在的机器的存储体系中,reduce任务很可能不在同一台机器上运行,所以需要远程下载map任务的中间输出。ShuffleClient不仅将shuffle文件上传到其他Executor或者下载到本地的客户端,也提供了可以被其他Executor访问的shuffle服务。
在BlockManager中创建ShuffleClinet代码如下:
// Client to read other executors' shuffle files. This is either an external service, or just the
// standard BlockTransferService to directly connect to other Executors.
private[spark] val shuffleClient = if (externalShuffleServiceEnabled) {
val transConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores)
new ExternalShuffleClient(transConf, securityManager, securityManager.isAuthenticationEnabled(),
securityManager.isSaslEncryptionEnabled())
} else {
blockTransferService
}
从上面的代码中可以到,当有外部的ShuffleClient时,新建ExternalShuffleClient,否则默认为BlockTransferService。BlockTransferService必须在调用init方法后才能提供服务。BlockTransferService是一个抽象类,实现它的子类是NettyBlockTransferService,其初始化步骤:
1. 创建RpcServer(实际是其子类NettyBlockRpcServer);
2. 创建TransportContext;
3. 创建Rpc 客户端工厂 TransportClientFactory;
4. 创建Netty服务器 TransportServer,可以修改属性spark.blockManager.port改变TransportServer的端口
override def init(blockDataManager: BlockDataManager): Unit = {
val rpcHandler = new NettyBlockRpcServer(conf.getAppId, serializer, blockDataManager)
var serverBootstrap: Option[TransportServerBootstrap] = None
var clientBootstrap: Option[TransportClientBootstrap] = None
if (authEnabled) {
serverBootstrap = Some(new SaslServerBootstrap(transportConf, securityManager))
clientBootstrap = Some(new SaslClientBootstrap(transportConf, conf.getAppId, securityManager,
securityManager.isSaslEncryptionEnabled()))
}
transportContext = new TransportContext(transportConf, rpcHandler)
clientFactory = transportContext.createClientFactory(clientBootstrap.toSeq.asJava)
server = createServer(serverBootstrap.toList)
appId = conf.getAppId
logInfo("Server created on " + server.getPort)
}
1 Block的RPC服务 NettyBlockRpcServer
当map任务与reduce任务处于不同的节点是,reduce任务需要从远端节点下载map任务的中间输出,因此NettyBlockRpcServer提供