- 博客(4)
- 资源 (3)
- 收藏
- 关注
RSS订阅原创 spar 内存存储 MemoryStore
MemoryStorespark将内存中的block抽象为MemoryEntry//spark将内存中的block抽象为MemoryEntryprivate sealed trait MemoryEntry[T] { def size: Long//当前块的大小 def memoryMode: MemoryMode//block存入内存的内存模式 def classTag: Cl...
2019-05-03 00:41:40
378
原创 spark 存储之磁盘存储
spark 存储之磁盘存储spark的数据需要写到磁盘上,主要有两个类负责这个工作.DiskBlockManager和DiskStoreDiskBlockManagerCreates and maintains the logical mapping between logical blocks and physical on-disk locations. One block is map...
2019-05-02 16:42:47
1008
原创 spark 内存管理模型
MemoryManagerAn abstract memory manager that enforces how memory is shared between execution and storage. In this context, execution memory refers that used for computation in shuffles, joins, sorts ...
2019-05-02 02:13:23
279
原创 从spark学scala语法 1
with mutable.MultiMap[TaskAttemptId, BlockId]//BlockInfoManager 类 /** * Tracks the set of blocks that each task has locked for writing. */ @GuardedBy("this") private[this] val writeLocksB...
2019-05-01 01:13:36
145
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人