Spark 内存管理之StaticMemoryManager

概要

Spark 内存管理概述 介绍了内存管理接口MemoryManager对内存的大致划分,这篇研究其子类StaticMemoryManager,静态内存管理器,的行为,主要是各部分内存的占比。StaticMemoryManager是Spark 1.6之前唯一的内存管理器。

StaticMemoryManager

Spark 内存管理概述 介绍了MemoryManager的UML,StaticMemoryManager继承自MemoryManager,如下 

StaticMemoryManager分别使用ExecutionMemoryPool、StorageMemoryPool管理execution、storage、unroll内存,UML如下 

内存管理

回顾MemoryManager

MemoryManager将内存划分为如下几部分,接下来研究静态内存管理中下面各部分占比 
这里写图片描述

静态内存管理

之所以称为静态内存管理,是因为storage、execution内存占比和界限是固定的。看下代码中对storage、execution、unroll内存大小的限制

  1. storage 
  2. execution 
  3. unroll 

总结如下

storage 0.6 storage-safety 0.6 * 0.9 * (1.0-0.2)
unroll 0.6 * 0.9 * 0.2
storage-reserved 0.6 * 0.1
execution 0.2 execution-safety 0.2 * 0.8
execution-reserved 0.2 * 0.2
other 0.2  

饼图表示如下 

因为无法准确计算实际使用中的内存,因此,storage、execution都会保留一部分内存,就是上图中的storage-reservedexecution-reserved

设置

spark 1.6版本之后默认使用统一资源管理器,由参数spark.memory.useLegacyMode控制,如下 

总结

介绍StaticMemoryManager的结构,及其管理的内存storage、execution、unroll等占比,这里所说的管理只是数值的维护。


(完)































https://www.douban.com/search?q=%e7%8e%a9%e5%ae%b6%e6%b1%87%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=ABC%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e9%87%91%e5%b0%8a%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%90%8d%e7%88%b5%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%bc%9f%e5%be%b3%e4%ba%9a%e6%b4%b2%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e7%89%9b%e5%bd%a9%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a9%e9%9f%b3%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a9%e4%ba%91%e9%98%81%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449

http://home.babytree.com/u12992707376
http://home.babytree.com/u12992707376/journal/show/14230250
http://www.hdb.com/article/cycrb.html
http://www.hdb.com/article/aqcrb.html
http://www.hdb.com/article/tqcrb.html
http://www.hdb.com/article/eqcrb.html
http://www.hdb.com/article/jqcrb.html
http://www.hdb.com/article/6qcrb.html


http://www.hdb.com/article/uccrb.html
http://www.hdb.com/article/accrb.html
http://www.hdb.com/article/7ccrb.html
http://www.hdb.com/article/eccrb.html
http://www.hdb.com/article/jccrb.html
http://www.hdb.com/article/8ccrb.html
http://www.hdb.com/article/lccrb.html
http://www.hdb.com/article/4ccrb.html
http://www.hdb.com/article/qccrb.html
http://www.hdb.com/article/cccrb.html
http://www.hdb.com/article/gccrb.html
http://www.hdb.com/article/zccrb.html
http://www.hdb.com/article/9ccrb.html
http://www.hdb.com/article/wccrb.html
http://www.hdb.com/article/sccrb.html
http://www.hdb.com/article/0gcrb.html
http://www.hdb.com/article/2gcrb.html
http://www.hdb.com/article/agcrb.html
http://www.hdb.com/article/5gcrb.html
http://www.hdb.com/article/7gcrb.html
http://www.hdb.com/article/tgcrb.html
http://www.hdb.com/article/pgcrb.html
http://www.hdb.com/article/egcrb.html
http://www.hdb.com/article/fgcrb.html
http://www.hdb.com/article/sgcrb.html
http://www.hdb.com/article/uzcrb.html
http://www.hdb.com/article/kzcrb.html
http://www.hdb.com/article/ezcrb.html
http://www.hdb.com/article/6zcrb.html
http://www.hdb.com/article/ozcrb.html
http://www.hdb.com/article/rzcrb.html
http://www.hdb.com/article/8zcrb.html
http://www.hdb.com/article/yzcrb.html




http://www.hdb.com/article/gzcrb.html
http://www.hdb.com/article/zzcrb.html
http://www.hdb.com/article/izcrb.html
http://www.hdb.com/article/1zcrb.html
http://www.hdb.com/article/hzcrb.html
http://www.hdb.com/article/szcrb.html
http://www.hdb.com/article/0icrb.html
http://www.hdb.com/article/2icrb.html
http://www.hdb.com/article/aicrb.html
http://www.hdb.com/article/5icrb.html
http://www.hdb.com/article/ticrb.html
http://www.hdb.com/article/picrb.html




http://www.hdb.com/article/gicrb.html
http://www.hdb.com/article/b1crb.html
http://www.hdb.com/article/21crb.html
http://www.hdb.com/article/51crb.html
http://www.hdb.com/article/t1crb.html
http://www.hdb.com/article/p1crb.html
http://www.hdb.com/article/k1crb.html
http://www.hdb.com/article/e1crb.html
http://www.hdb.com/article/31crb.html
http://www.hdb.com/article/j1crb.html
http://www.hdb.com/article/61crb.html




http://www.hdb.com/article/z1crb.html
http://www.hdb.com/article/11crb.html
http://www.hdb.com/article/h1crb.html
http://www.hdb.com/article/91crb.html
http://www.hdb.com/article/d1crb.html
http://www.hdb.com/article/w1crb.html
http://www.hdb.com/article/f1crb.html
http://www.hdb.com/article/s1crb.html
http://www.hdb.com/article/x1crb.html
http://www.hdb.com/article/uhcrb.html
http://www.hdb.com/article/2hcrb.html


http://www.hdb.com/article/7hcrb.html
http://www.hdb.com/article/thcrb.html
http://www.hdb.com/article/phcrb.html
http://www.hdb.com/article/khcrb.html
http://www.hdb.com/article/ehcrb.html
http://www.hdb.com/article/jhcrb.html
http://www.hdb.com/article/ohcrb.html
http://www.hdb.com/article/rhcrb.html
http://www.hdb.com/article/8hcrb.html
http://www.hdb.com/article/lhcrb.html
http://www.hdb.com/article/mhcrb.html
http://www.hdb.com/article/yhcrb.html


http://www.hdb.com/article/ghcrb.html
http://www.hdb.com/article/zhcrb.html
http://www.hdb.com/article/ihcrb.html
http://www.hdb.com/article/1hcrb.html
http://www.hdb.com/article/hhcrb.html
http://www.hdb.com/article/vhcrb.html
http://www.hdb.com/article/9hcrb.html
http://www.hdb.com/article/dhcrb.html
http://www.hdb.com/article/whcrb.html
http://www.hdb.com/article/fhcrb.html
http://www.hdb.com/article/shcrb.html
http://www.hdb.com/article/xhcrb.html


http://www.hdb.com/article/nvcrb.html
http://www.hdb.com/article/avcrb.html
http://www.hdb.com/article/5vcrb.html
http://www.hdb.com/article/7vcrb.html
http://www.hdb.com/article/tvcrb.html
http://www.hdb.com/article/pvcrb.html
http://www.hdb.com/article/kvcrb.html
http://www.hdb.com/article/evcrb.html
http://www.hdb.com/article/3vcrb.html
http://www.hdb.com/article/jvcrb.html
http://www.hdb.com/article/6vcrb.html
http://www.hdb.com/article/ovcrb.html
http://www.hdb.com/article/8vcrb.html




http://www.hdb.com/article/b9crb.html
http://www.hdb.com/article/29crb.html
http://www.hdb.com/article/a9crb.html
http://www.hdb.com/article/59crb.html
http://www.hdb.com/article/79crb.html
http://www.hdb.com/article/t9crb.html
http://www.hdb.com/article/p9crb.html
http://www.hdb.com/article/k9crb.html
http://www.hdb.com/article/39crb.html
http://www.hdb.com/article/j9crb.html
http://www.hdb.com/article/69crb.html
http://www.hdb.com/article/o9crb.html
http://www.hdb.com/article/r9crb.html


http://www.hdb.com/article/49crb.html
http://www.hdb.com/article/q9crb.html
http://www.hdb.com/article/g9crb.html
http://www.hdb.com/article/z9crb.html
http://www.hdb.com/article/i9crb.html
http://www.hdb.com/article/19crb.html
http://www.hdb.com/article/h9crb.html
http://www.hdb.com/article/v9crb.html
http://www.hdb.com/article/d9crb.html
http://www.hdb.com/article/w9crb.html
http://www.hdb.com/article/f9crb.html
http://www.hdb.com/article/s9crb.html
http://www.hdb.com/article/x9crb.html
http://www.hdb.com/article/udcrb.html
http://www.hdb.com/article/2dcrb.html
http://www.hdb.com/article/ndcrb.html
http://www.hdb.com/article/adcrb.html
http://www.hdb.com/article/5dcrb.html
http://www.hdb.com/article/7dcrb.html


http://www.hdb.com/article/edcrb.html
http://www.hdb.com/article/3dcrb.html
http://www.hdb.com/article/jdcrb.html
http://www.hdb.com/article/6dcrb.html
http://www.hdb.com/article/odcrb.html
http://www.hdb.com/article/rdcrb.html
http://www.hdb.com/article/ldcrb.html
http://www.hdb.com/article/mdcrb.html
http://www.hdb.com/article/ydcrb.html
http://www.hdb.com/article/4dcrb.html
http://www.hdb.com/article/qdcrb.html
http://www.hdb.com/article/cdcrb.html
http://www.hdb.com/article/gdcrb.html
http://www.hdb.com/article/zdcrb.html
http://www.hdb.com/article/idcrb.html
http://www.hdb.com/article/1dcrb.html


https://www.douban.com/search?q=%e4%bc%98%e6%b8%b8%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3-%e5%ae%98%e7%bd%91QQ61777449
https://www.douban.com/search?q=%e4%bc%98%e6%b8%b8%e5%a8%b1%e4%b9%90%e6%b3%a8%e5%86%8c%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e5%88%a9%e4%bf%a1%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%88%a9%e4%bf%a1%e5%b9%b3%e5%8f%b0%e6%b3%a8%e5%86%8c%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%bc%80%e6%88%b7%e6%b3%a8%e5%86%8cQQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e6%b3%a8%e5%86%8cQQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%ba%bf%e4%b8%87%e5%85%88%e7%94%9f%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%b8%9c%e6%a3%ae%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%b8%9c%e6%a3%ae%e5%b9%b3%e5%8f%b0%e6%b3%a8%e5%86%8c%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e5%8d%8e%e4%ba%ba%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e6%b8%b8%e6%88%8f%e5%bc%80%e6%88%b7%e6%b3%a8%e5%86%8cQQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3%e5%ae%98%e7%bd%91QQ61777449
https://www.douban.com/search?q=%e4%bc%98%e5%8d%9a%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a7%e5%9c%a3%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a7%e5%9c%a3%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3%e5%ae%98%e7%bd%91QQ61777449
https://www.douban.com/search?q=%e7%99%be%e5%88%a9%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%bf%9d%e6%97%b6%e6%8d%b7%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%8e%8c%e4%b8%ad%e5%bd%a9%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%b8%9d%e8%8b%91club%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%ac%a2%e4%b9%90%e5%90%a7%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e9%87%91%e6%b2%99%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449


























评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值