spark优化之临时目录

最新推荐文章于 2022-11-12 14:25:10 发布

weixin_30355437

最新推荐文章于 2022-11-12 14:25:10 发布

阅读量126

点赞数

文章标签：大数据

原文链接：http://www.cnblogs.com/hark0623/p/4512506.html

版权

官方是这样说的：

Directory to use for "scratch" space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks. NOTE: In Spark 1.0 and later this will be overriden by SPARK_LOCAL_DIRS (Standalone, Mesos) or LOCAL_DIRS (YARN) environment variables set by the cluster manager.

大概就是MAP和RDD的临时数据可能存放在spark.local.dir（SPARK_LOCAL_DIRS）指定的目录中，默认是在/tmp中。

该目录的磁盘一定是快速的本地磁盘系统，我们可以考虑把这个目录来单独放成一个磁盘，或者以都好隔开放入多个磁盘，来提高i/o读写速度

转载于:https://www.cnblogs.com/hark0623/p/4512506.html