【源码解读系列之spark】Spark RDD缓存函数cache() 和 persist()原理和使用

最新推荐文章于 2023-11-16 13:28:04 发布

Data_IT_Farmer

最新推荐文章于 2023-11-16 13:28:04 发布

阅读量722

点赞数

分类专栏： Spark 文章标签： spark rdd cache() persist 变量缓存

本文链接：https://blog.csdn.net/helloxiaozhe/article/details/103222990

版权

Spark 专栏收录该内容

44 篇文章 6 订阅

订阅专栏

1、RDD的缓存级别

顺便看一下RDD都有哪些缓存级别，查看 storagelevel.py代码StorageLevel类如下:

__all__ = ["StorageLevel"]


class StorageLevel(object):

    """
    Flags for controlling the storage of an RDD. Each StorageLevel records whether to use memory,
    whether to drop the RDD to disk if it falls out of memory, whether to keep the data in memory
    in a JAVA-specific serialized format, and whether to replicate the RDD partitions on multiple
    nodes. Also contains static constants for some commonly used storage levels, MEMORY_ONLY.
    Since the data is always serialized on the Python side, all the constants use the serialized
    formats.
    """

    def __init__(self, useDisk, useMemory, useOffHeap, deserialized, replication=1):
        self.useDisk = useDisk
        self.useMemory = useMemory
        self.useOffHeap = useOffHeap
        self.deserialized = deserialized
        self.replication = replication

    def __repr__(self):
        return "StorageLevel(%s, %s, %s, %s, %s)" % (
            self.useDisk, self.useMemory, self.useOffHeap, self.deserialized, self.replication)

    def __str__(self):
        result = ""
        result += "Disk " if self.useDisk else ""
        result += "Memory " if self.useMemory else ""
        result += "OffHeap " if self.useOffHeap else ""
        result += "Deserialized " if self.deserialized else "Serialized "
        result += "%sx Replicated" % self.replication
        return result

StorageLevel.DISK_ONLY = StorageLevel(True, False, False, False)
StorageLevel.DISK_ONLY_2 = StorageLevel(True, False, False, False, 2)
StorageLevel.MEMORY_ONLY = StorageLevel(False, True, False, False)
StorageLevel.MEMORY_ONLY_2 = StorageLevel(False, True, False, False, 2)
StorageLevel.MEMORY_AND_DISK = StorageLevel(True, True, False, False)
StorageLevel.MEMORY_AND_DISK_2 = StorageLevel(True, True, False, False, 2)
StorageLevel.OFF_HEAP = StorageLevel(True, True, True, False, 1)

"""
.. note:: The following four storage level constants are deprecated in 2.0, since the records
    will always be serialized in Python.
"""
StorageLevel.MEMORY_ONLY_SER = StorageLevel.MEMORY_ONLY
""".. note:: Deprecated in 2.0, use ``StorageLevel.MEMORY_ONLY`` instead."""
StorageLevel.MEMORY_ONLY_SER_2 = StorageLevel.MEMORY_ONLY_2
""".. note:: Deprecated in 2.0, use ``StorageLevel.MEMORY_ONLY_2`` instead."""
StorageLevel.MEMORY_AND_DISK_SER = StorageLevel.MEMORY_AND_DISK
""".. note:: Deprecated in 2.0, use ``StorageLevel.MEMORY_AND_DISK`` instead."""
StorageLevel.MEMORY_AND_DISK_SER_2 = StorageLevel.MEMORY_AND_DISK_2
""".. note:: Deprecated in 2.0, use ``StorageLevel.MEMORY_AND_DISK_2`` instead."""

可以看到StorageLevel类的主构造器包含了5个参数：

useDisk：使用硬盘（外存）
useMemory：使用内存
useOffHeap：使用堆外内存，这是Java虚拟机里面的概念，堆外内存意味着把内存对象分配在Java虚拟机的堆以外的内存，这些内存直接受操作系统管理（而不是虚拟机）。这样就能够保持一个较小的堆，以减少垃圾收集对应用的影响。
deserialized：反序列化，其逆过程序列化（Serialization）是java提供的一种机制，将对象表示成一连串的字节；而反序列化就表示将字节恢复为对象的过程。序列化是对象永久化的一种机制，可以将对象及其属性保存起来，并能在反序列化后直接恢复这个对象。序列化方式存储对象可以节省磁盘或内存的空间，一般序列化：反序列化=1:3
replication：备份数（在多个节点上备份）
理解了这5个参数，StorageLevel 的12种缓存级别就不难理解了。

比如，StorageLevel.MEMORY_AND_DISK_2 = StorageLevel(True, True, False, False, 2) 就表示使用这种缓存级别的RDD将存储在硬盘以及内存中，使用序列化（在硬盘中），并且在多个节点上备份2份（正常的RDD只有一份）

另外还注意到有一种特殊的缓存级别

StorageLevel.OFF_HEAP = StorageLevel(True, True, True, False, 1)

使用了堆外内存，StorageLevel 类的源码中有一段代码可以看出这个的特殊性，它不能和其它几个参数共存。

2、Spark RDD Cache

允许将RDD缓存到内存中或磁盘上，以便于重用
Spark提供了多种缓存级别，以便于用户根据实际需求进行调整
Spark RDD cache/persist

3、RDD cache使用

test_rdd = sc.textFile(“hdfs://nn:8020/input”)

test_rdd.cache() #实际上是test_rdd.persist(StorageLevel.MEMORY_ONLY)

test_rdd.unpersist() # 删除缓存变量: Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

4、cache和persist的区别

基于Spark源码，可以看到

    def cache(self):
        """
        Persist this RDD with the default storage level (C{MEMORY_ONLY}).
        """
        self.is_cached = True
        self.persist(StorageLevel.MEMORY_ONLY)
        return self

说明是cache()调用了persist(), 想要知道二者的不同还需要看一下persist函数：

    def cache(self):
        """
        Persist this RDD with the default storage level (C{MEMORY_ONLY}).
        """
        self.is_cached = True
        self.persist(StorageLevel.MEMORY_ONLY)
        return self

可以看到persist()内部调用了persist(StorageLevel.MEMORY_ONLY)，继续深入：

    def persist(self, storageLevel=StorageLevel.MEMORY_ONLY):
        """
        Set this RDD's storage level to persist its values across operations
        after the first time it is computed. This can only be used to assign
        a new storage level if the RDD does not have a storage level set yet.
        If no storage level is specified defaults to (C{MEMORY_ONLY}).

        >>> rdd = sc.parallelize(["b", "a", "c"])
        >>> rdd.persist().is_cached
        True
        """
        self.is_cached = True
        javaStorageLevel = self.ctx._getJavaStorageLevel(storageLevel)
        self._jrdd.persist(javaStorageLevel)
        return self

可以看出来persist有一个 StorageLevel 类型的参数，这个表示的是RDD的缓存级别。

至此便可得出cache和persist的区别了：cache只有一个默认的缓存级别MEMORY_ONLY ，而persist可以根据情况设置其它的缓存级别。

5、删除缓存变量

可以采用 : test_rdd.unpersist()

    def unpersist(self):
        """
        Mark the RDD as non-persistent, and remove all blocks for it from
        memory and disk.
        """
        self.is_cached = False
        self._jrdd.unpersist()
        return self

参考: https://blog.csdn.net/leehbing/article/details/78866497