BlockManager

Spark为了避免Hadoop读写磁盘的I/O操作成为性能瓶颈,优先将配置信息、中间计算结果等数据存入内存,极大的提高了系统的执行效率。除此之外,还可以将这些数据放入磁盘或者外部存储系统中。

1. 块管理器BlockManager 的构造

块管理器BlockManager是Spark存储体系中的核心组件。Driver 和 Executor都会创建BlockManager。其主构造器如下:

/**
 * Manager running on every node (driver and executors) which provides interfaces for putting and
 * retrieving blocks both locally and remotely into various stores (memory, disk, and off-heap).
 *
 * Note that #initialize() must be called before the BlockManager is usable.
 */
private[spark] class BlockManager(
    executorId: String,
    rpcEnv: RpcEnv,
    val master: BlockManagerMaster,
    defaultSerializer: Serializer,
    val conf: SparkConf,
    memoryManager: MemoryManager,
    mapOutputTracker: MapOutputTracker,
    shuffleManager: ShuffleManager,
    blockTransferService: BlockTransferService,
    securityManager: SecurityManager,
    numUsableCores: Int)
  extends BlockDataManager with Logging

BlockManager主要组成:
0. BlockManagerMaster:Driver上的BlockManagerMaster对存在于Executor上的BlockManager统一管理
1. DiskBlockManager:磁盘块管理器
2. blockInfo:用于缓存BlockId和对应的BlockInfo;
3. ExecutionContext: 创建ExecutionContext,它是以ThreadPoolExecutor线程池作为服务的,每个线程的名称前缀是block-manager-future,最大可以创建128个线程;
4. MemoryStore: 内存存储,将blocks存储在内存中,存储方式可以是Java object数组或者序列化后的
ByteBuffers

5. DiskStore:磁盘存储
6. ExternalBlockStore : ExternalBlockStore 存储BlockManager blocks,内部实际使用的是TachyonBlockManager进行管理
7. ShuffleClient:shuffle客户端ShuffleClient,默认使用BlockTransferService ,通过spark.shuffle.service.enabled属性设置为true则可以使用外部的ShuffleService
8. BlockManagerSlaveEndpoint:注册BlockManagerSlaveEndpoint并且返回它的引用(默认Netty模式的话为NettyRpcEndpointRef);
9. metadataCleaner : 非广播Block清理器;
10. broadcastCleaner : 广播Block清理器;
11. CompressionCodec :压缩算法实现.

  val diskBlockManager = new DiskBlockManager(this, conf)
  private val blockInfo = new TimeStampedHashMap[BlockId, BlockInfo]

  private val futureExecutionContext = ExecutionContext.fromExecutorService(  ThreadUtils.newDaemonCachedThreadPool("block-manager-future", 128))

  // Actual storage of where blocks are kept
  private var externalBlockStoreInitialized = false
  private[spark] val memoryStore = new MemoryStore(this, memoryManager)
  private[spark] val diskStore = new DiskStore(this, diskBlockManager)
  private[spark] lazy val externalBlockStore: ExternalBlockStore = {
    externalBlockStoreInitialized = true
    new ExternalBlockStore(this, executorId)
  }
  memoryManager.setMemoryStore(memoryStore)

  private[spark]
  val externalShuffleServiceEnabled = conf.getBoolean("spark.shuffle.service.enabled", false)
  // Client to read other executors' shuffle files. This is either an external service, or just the
  // standard BlockTransferService to directly connect to other Executors.
  private[spark] val shuffleClient = if (externalShuffleServiceEnabled) {
    val transConf = SparkTransportConf.fromSparkConf(conf, "shuffle", numUsableCores)
    new ExternalShuffleClient(transConf, securityManager, securityManager.isAuthenticationEnabled(),
      securityManager.isSaslEncryptionEnabled())
  } else {
    blockTransferService
  }

  // Register a [[RpcEndpoint]] with a name and return its [[RpcEndpointRef]].
  private val slaveEndpoint = rpcEnv.setupEndpoint(
    "BlockManagerEndpoint" + BlockManager.ID_GENERATOR.next,
    new BlockManagerSlaveEndpoint(rpcEnv, this, mapOutputTracker))

  private val metadataCleaner = new MetadataCleaner(
    MetadataCleanerType.BLOCK_MANAGER, this.dropOldNonBroadcastBlocks, conf)
  private val broadcastCleaner = new MetadataCleaner(
    MetadataCleanerType.BROADCAST_VARS, this.dropOldBroadcastBlocks, conf)

  /* The compression codec to use. Note that the "lazy" val is necessary because we want to delay
   * the initialization of the compression codec until it is first used. The reason is that a Spark
   * program could be using a user-defined codec in a third party jar, which is loaded in
   * Executor.updateDependencies. When the BlockManager is initialized, user level jars hasn't been
   * loaded yet. */
  private lazy val compressionCodec: CompressionCodec = CompressionCodec.createCodec(conf)

2. BlockManager初始化

BlockManager要生效,必须进行初始化操作。而且不能在BlockManager构造过程中进行初始化。因为这个时候应用程序的ID可能还没获得。

  
  • 2
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值