Spark 内存管理之StaticMemoryManager

最新推荐文章于 2023-04-27 13:22:34 发布

jindg1980

最新推荐文章于 2023-04-27 13:22:34 发布

阅读量937

点赞数

概要

Spark 内存管理概述介绍了内存管理接口MemoryManager对内存的大致划分，这篇研究其子类StaticMemoryManager，静态内存管理器，的行为，主要是各部分内存的占比。StaticMemoryManager是Spark 1.6之前唯一的内存管理器。

StaticMemoryManager

Spark 内存管理概述介绍了MemoryManager的UML，StaticMemoryManager继承自MemoryManager，如下

StaticMemoryManager分别使用ExecutionMemoryPool、StorageMemoryPool管理execution、storage、unroll内存，UML如下

内存管理

回顾MemoryManager

MemoryManager将内存划分为如下几部分，接下来研究静态内存管理中下面各部分占比
这里写图片描述

静态内存管理

之所以称为静态内存管理，是因为storage、execution内存占比和界限是固定的。看下代码中对storage、execution、unroll内存大小的限制

storage
execution
unroll

总结如下

storage	0.6	storage-safety	0.6 * 0.9 * (1.0-0.2)
		unroll	0.6 * 0.9 * 0.2
		storage-reserved	0.6 * 0.1
execution	0.2	execution-safety	0.2 * 0.8
execution	0.2	execution-reserved	0.2 * 0.2
other	0.2

饼图表示如下

因为无法准确计算实际使用中的内存，因此，storage、execution都会保留一部分内存，就是上图中的storage-reserved、execution-reserved。

设置

spark 1.6版本之后默认使用统一资源管理器，由参数spark.memory.useLegacyMode控制，如下

总结

介绍StaticMemoryManager的结构，及其管理的内存storage、execution、unroll等占比，这里所说的管理只是数值的维护。

（完）

https://www.douban.com/search?q=%e7%8e%a9%e5%ae%b6%e6%b1%87%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=ABC%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e9%87%91%e5%b0%8a%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%90%8d%e7%88%b5%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%bc%9f%e5%be%b3%e4%ba%9a%e6%b4%b2%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e7%89%9b%e5%bd%a9%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a9%e9%9f%b3%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a9%e4%ba%91%e9%98%81%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449

http://home.babytree.com/u12992707376
http://home.babytree.com/u12992707376/journal/show/14230250
http://www.hdb.com/article/cycrb.html
http://www.hdb.com/article/aqcrb.html
http://www.hdb.com/article/tqcrb.html
http://www.hdb.com/article/eqcrb.html
http://www.hdb.com/article/jqcrb.html
http://www.hdb.com/article/6qcrb.html

http://www.hdb.com/article/uccrb.html
http://www.hdb.com/article/accrb.html
http://www.hdb.com/article/7ccrb.html
http://www.hdb.com/article/eccrb.html
http://www.hdb.com/article/jccrb.html
http://www.hdb.com/article/8ccrb.html
http://www.hdb.com/article/lccrb.html
http://www.hdb.com/article/4ccrb.html
http://www.hdb.com/article/qccrb.html
http://www.hdb.com/article/cccrb.html
http://www.hdb.com/article/gccrb.html
http://www.hdb.com/article/zccrb.html
http://www.hdb.com/article/9ccrb.html
http://www.hdb.com/article/wccrb.html
http://www.hdb.com/article/sccrb.html
http://www.hdb.com/article/0gcrb.html
http://www.hdb.com/article/2gcrb.html
http://www.hdb.com/article/agcrb.html
http://www.hdb.com/article/5gcrb.html
http://www.hdb.com/article/7gcrb.html
http://www.hdb.com/article/tgcrb.html
http://www.hdb.com/article/pgcrb.html
http://www.hdb.com/article/egcrb.html
http://www.hdb.com/article/fgcrb.html
http://www.hdb.com/article/sgcrb.html
http://www.hdb.com/article/uzcrb.html
http://www.hdb.com/article/kzcrb.html
http://www.hdb.com/article/ezcrb.html
http://www.hdb.com/article/6zcrb.html
http://www.hdb.com/article/ozcrb.html
http://www.hdb.com/article/rzcrb.html
http://www.hdb.com/article/8zcrb.html
http://www.hdb.com/article/yzcrb.html

http://www.hdb.com/article/gzcrb.html
http://www.hdb.com/article/zzcrb.html
http://www.hdb.com/article/izcrb.html
http://www.hdb.com/article/1zcrb.html
http://www.hdb.com/article/hzcrb.html
http://www.hdb.com/article/szcrb.html
http://www.hdb.com/article/0icrb.html
http://www.hdb.com/article/2icrb.html
http://www.hdb.com/article/aicrb.html
http://www.hdb.com/article/5icrb.html
http://www.hdb.com/article/ticrb.html
http://www.hdb.com/article/picrb.html

http://www.hdb.com/article/gicrb.html
http://www.hdb.com/article/b1crb.html
http://www.hdb.com/article/21crb.html
http://www.hdb.com/article/51crb.html
http://www.hdb.com/article/t1crb.html
http://www.hdb.com/article/p1crb.html
http://www.hdb.com/article/k1crb.html
http://www.hdb.com/article/e1crb.html
http://www.hdb.com/article/31crb.html
http://www.hdb.com/article/j1crb.html
http://www.hdb.com/article/61crb.html

http://www.hdb.com/article/z1crb.html
http://www.hdb.com/article/11crb.html
http://www.hdb.com/article/h1crb.html
http://www.hdb.com/article/91crb.html
http://www.hdb.com/article/d1crb.html
http://www.hdb.com/article/w1crb.html
http://www.hdb.com/article/f1crb.html
http://www.hdb.com/article/s1crb.html
http://www.hdb.com/article/x1crb.html
http://www.hdb.com/article/uhcrb.html
http://www.hdb.com/article/2hcrb.html

http://www.hdb.com/article/7hcrb.html
http://www.hdb.com/article/thcrb.html
http://www.hdb.com/article/phcrb.html
http://www.hdb.com/article/khcrb.html
http://www.hdb.com/article/ehcrb.html
http://www.hdb.com/article/jhcrb.html
http://www.hdb.com/article/ohcrb.html
http://www.hdb.com/article/rhcrb.html
http://www.hdb.com/article/8hcrb.html
http://www.hdb.com/article/lhcrb.html
http://www.hdb.com/article/mhcrb.html
http://www.hdb.com/article/yhcrb.html

http://www.hdb.com/article/ghcrb.html
http://www.hdb.com/article/zhcrb.html
http://www.hdb.com/article/ihcrb.html
http://www.hdb.com/article/1hcrb.html
http://www.hdb.com/article/hhcrb.html
http://www.hdb.com/article/vhcrb.html
http://www.hdb.com/article/9hcrb.html
http://www.hdb.com/article/dhcrb.html
http://www.hdb.com/article/whcrb.html
http://www.hdb.com/article/fhcrb.html
http://www.hdb.com/article/shcrb.html
http://www.hdb.com/article/xhcrb.html

http://www.hdb.com/article/nvcrb.html
http://www.hdb.com/article/avcrb.html
http://www.hdb.com/article/5vcrb.html
http://www.hdb.com/article/7vcrb.html
http://www.hdb.com/article/tvcrb.html
http://www.hdb.com/article/pvcrb.html
http://www.hdb.com/article/kvcrb.html
http://www.hdb.com/article/evcrb.html
http://www.hdb.com/article/3vcrb.html
http://www.hdb.com/article/jvcrb.html
http://www.hdb.com/article/6vcrb.html
http://www.hdb.com/article/ovcrb.html
http://www.hdb.com/article/8vcrb.html

http://www.hdb.com/article/b9crb.html
http://www.hdb.com/article/29crb.html
http://www.hdb.com/article/a9crb.html
http://www.hdb.com/article/59crb.html
http://www.hdb.com/article/79crb.html
http://www.hdb.com/article/t9crb.html
http://www.hdb.com/article/p9crb.html
http://www.hdb.com/article/k9crb.html
http://www.hdb.com/article/39crb.html
http://www.hdb.com/article/j9crb.html
http://www.hdb.com/article/69crb.html
http://www.hdb.com/article/o9crb.html
http://www.hdb.com/article/r9crb.html

http://www.hdb.com/article/49crb.html
http://www.hdb.com/article/q9crb.html
http://www.hdb.com/article/g9crb.html
http://www.hdb.com/article/z9crb.html
http://www.hdb.com/article/i9crb.html
http://www.hdb.com/article/19crb.html
http://www.hdb.com/article/h9crb.html
http://www.hdb.com/article/v9crb.html
http://www.hdb.com/article/d9crb.html
http://www.hdb.com/article/w9crb.html
http://www.hdb.com/article/f9crb.html
http://www.hdb.com/article/s9crb.html
http://www.hdb.com/article/x9crb.html
http://www.hdb.com/article/udcrb.html
http://www.hdb.com/article/2dcrb.html
http://www.hdb.com/article/ndcrb.html
http://www.hdb.com/article/adcrb.html
http://www.hdb.com/article/5dcrb.html
http://www.hdb.com/article/7dcrb.html

http://www.hdb.com/article/edcrb.html
http://www.hdb.com/article/3dcrb.html
http://www.hdb.com/article/jdcrb.html
http://www.hdb.com/article/6dcrb.html
http://www.hdb.com/article/odcrb.html
http://www.hdb.com/article/rdcrb.html
http://www.hdb.com/article/ldcrb.html
http://www.hdb.com/article/mdcrb.html
http://www.hdb.com/article/ydcrb.html
http://www.hdb.com/article/4dcrb.html
http://www.hdb.com/article/qdcrb.html
http://www.hdb.com/article/cdcrb.html
http://www.hdb.com/article/gdcrb.html
http://www.hdb.com/article/zdcrb.html
http://www.hdb.com/article/idcrb.html
http://www.hdb.com/article/1dcrb.html

https://www.douban.com/search?q=%e4%bc%98%e6%b8%b8%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3-%e5%ae%98%e7%bd%91QQ61777449
https://www.douban.com/search?q=%e4%bc%98%e6%b8%b8%e5%a8%b1%e4%b9%90%e6%b3%a8%e5%86%8c%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e5%88%a9%e4%bf%a1%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%88%a9%e4%bf%a1%e5%b9%b3%e5%8f%b0%e6%b3%a8%e5%86%8c%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%bc%80%e6%88%b7%e6%b3%a8%e5%86%8cQQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e6%b3%a8%e5%86%8cQQ61777449
https://www.douban.com/search?q=%e5%8d%9a%e7%8c%ab%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%ba%bf%e4%b8%87%e5%85%88%e7%94%9f%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%b8%9c%e6%a3%ae%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%b8%9c%e6%a3%ae%e5%b9%b3%e5%8f%b0%e6%b3%a8%e5%86%8c%e5%bc%80%e6%88%b7QQ61777449
https://www.douban.com/search?q=%e5%8d%8e%e4%ba%ba%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e6%b8%b8%e6%88%8f%e5%bc%80%e6%88%b7%e6%b3%a8%e5%86%8cQQ61777449
https://www.douban.com/search?q=%e6%98%93%e8%b4%ad%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3%e5%ae%98%e7%bd%91QQ61777449
https://www.douban.com/search?q=%e4%bc%98%e5%8d%9a%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a7%e5%9c%a3%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%a4%a7%e5%9c%a3%e6%b8%b8%e6%88%8f%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3%e5%ae%98%e7%bd%91QQ61777449
https://www.douban.com/search?q=%e7%99%be%e5%88%a9%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e4%bf%9d%e6%97%b6%e6%8d%b7%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%8e%8c%e4%b8%ad%e5%bd%a9%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e5%b8%9d%e8%8b%91club%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e6%ac%a2%e4%b9%90%e5%90%a7%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449
https://www.douban.com/search?q=%e9%87%91%e6%b2%99%e5%a8%b1%e4%b9%90%e5%b9%b3%e5%8f%b0%e6%80%bb%e4%bb%a3QQ61777449