IMG-GPU之内存管理

Victor Fung

已于 2024-05-07 16:04:30 修改

阅读量97

点赞数

分类专栏： GPU领域文章标签： GPU

于 2022-05-13 16:52:17 首次发布

本文链接：https://blog.csdn.net/fs3296/article/details/124753115

版权

GPU领域专栏收录该内容

1 篇文章 0 订阅

订阅专栏

IMG-GPU之内存管理

引言
1 物理内存管理
2 虚拟内存管理
3 具体实现

引言

IMG-GPU对内存划分为两个层次，底层负责具体的物理内存的管理；基于物理内存管理层构建上层的不同用户的应用内存，或者说虚拟内存管理。

参考源码地址: https://gitlab.freedesktop.org/frankbinns/powervr/-/tree/ddk/1.17@6210866/drivers/gpu/drm/img-rogue

1 物理内存管理

1.1 物理内存划分

在struct _PVRSRV_DEVICE_NODE_中定义了不同的物理内存heap：

typedef struct _PVRSRV_DEVICE_NODE_
{
    # 略略略... ...
	PHYS_HEAP              *psMMUPhysHeap;
    # 略略略... ...
    PHYS_HEAP               *psFWMainPhysHeap;
	PHYS_HEAP               *psFWCfgPhysHeap;
	PHYS_HEAP               *apsFWPremapPhysHeap[RGX_NUM_OS_SUPPORTED];
	# 略略略... ...
	PHYS_HEAP				**papsRegisteredPhysHeaps;
	PHYS_HEAP				*apsPhysHeap[PVRSRV_PHYS_HEAP_LAST];
	# 略略略... ...
#if defined(SUPPORT_AUTOVZ)
	/* Phys Heap reserved for storing the MMU mappings of firmware.
	 * The memory backing up this Phys Heap must persist between driver or OS reboots */
	PHYS_HEAP               *psFwMMUReservedPhysHeap;
#endif
	# 略略略... ...
};

上述heap分为两类：一类称为原始heap，也就是通过PHYS_HEAP_CONFIG创建出来的；另一类就是heap拷贝，它来自于原始heap的备份。

1.2 原始物理heap

原始物理heap包括：

papsRegisteredPhysHeaps: 系统初始配置的所有heap；
psFWMainPhysHeap：固件FW_MAIN区域；
psFWCfgPhysHeap：固件FW_CONFIG区域；
psFwMMUReservedPhysHeap：自动虚拟化条件下，映射固件的MMU页表分配区域；
apsFWPremapPhysHeap：虚拟化条件下，每个guest的固件区域；

注：上述的heap通过g_psPhysHeapList串成一个链表，具体细节在函数PhysHeapCreate中。

1.2.1 heap配置

heap配置是papsRegisteredPhysHeaps的构建参数，由系统初始化指定。

typedef struct _PHYS_HEAP_CONFIG_
{
	PHYS_HEAP_TYPE        eType;                /*!< Class of heap and PMR factory used */
	IMG_CHAR*             pszPDumpMemspaceName; /*!< Name given to the heap's symbolic memory
	                                                 space in a PDUMP enabled driver */
	PHYS_HEAP_FUNCTIONS*  psMemFuncs;           /*!< Physical address translation functions */

	IMG_CPU_PHYADDR       sStartAddr;           /*!< CPU Physical base address of memory region */
	IMG_DEV_PHYADDR       sCardBase;            /*!< Device physical base address of memory
	                                                 region as seen from the PoV of the GPU */
	IMG_UINT64            uiSize;               /*!< Size of memory region in bytes */

	IMG_HANDLE            hPrivData;            /*!< System layer private data shared with
	                                                 psMemFuncs */

	PHYS_HEAP_USAGE_FLAGS ui32UsageFlags;       /*!< Supported uses flags, conveys the type of
	                                                 buffers the physical heap can be used for */
} PHYS_HEAP_CONFIG;

在目前的DDK-1.17开源驱动中，只提供了PHYS_HEAP_TYPE_UMA类型的heap，该heap是由OS管理物理页；此外还有PHYS_HEAP_TYPE_LMA和PHYS_HEAP_TYPE_DMA，这两个heap由DDK驱动管理物理页，也就是GPU专用物理区域。

1.2.2 heap构建

首先，在函数PhysHeapCreateDeviceHeapsFromConfigs中根据PHYS_HEAP_CONFIG构建papsRegisteredPhysHeaps；
然后，在函数RGXPhysMemDeviceHeapsInit中，如果heap_config配置了PHYS_HEAP_USAGE_FW_MAIN，则构建psFWMainPhysHeap和psFWCfgPhysHeap；如果heap_config配置了PHYS_HEAP_USAGE_FW_MAIN且在自动虚拟化模式的HOST下，则构建psFwMMUReservedPhysHeap；
HOST驱动会在全虚拟化模式下的RGXInitCreateFWKernelMemoryContext函数或半虚拟化模式下的PvzServerMapDevPhysHeap中，通过调用函数RGXFwRawHeapAllocMap构建每个guest的固件heap并存放在apsFWPremapPhysHeap中。

1.3 物理heap备份

struct _PVRSRV_DEVICE_NODE_中的psMMUPhysHeap和apsPhysHeap是备份heap：

前者查询原始物理heap中拷贝PVRSRV_PHYS_HEAP_GPU_LOCAL类型的heap指针，该heap是MMU页表占用的物理页默认的分配heap，特别地，在自动虚拟化模式下，固件上下文的MMU页表是psFwMMUReservedPhysHeap；
后者在构建原始heap过程中，会拷贝相关heap到数组中，拷贝位置为相关heap的flag类型，该heap是一个数组，建立相关heap的flag位到heap指针的索引，方便后续查询。

在拷贝heap过程中通过函数PhysHeapAcquireByDevPhysHeap从g_psPhysHeapList查询相关flag的heap，具体过程如下：

如果在g_psPhysHeapList找到相关heap，则返回结果，否则进入下一步。
在gasHeapProperties表中查找回退flag，按回退flag返回第一步重新查询，直到回退flag为0，表示查询失败。

注：回退表定义如下

static PHYS_HEAP_PROPERTIES gasHeapProperties[PVRSRV_PHYS_HEAP_LAST] =
{
	/* eFallbackHeap,               bPVRLayerAcquire, bUserModeAlloc */
    {  PVRSRV_PHYS_HEAP_DEFAULT,    IMG_TRUE,         IMG_TRUE  }, /* DEFAULT */
    {  PVRSRV_PHYS_HEAP_DEFAULT,    IMG_TRUE,         IMG_TRUE  }, /* GPU_LOCAL */
    {  PVRSRV_PHYS_HEAP_DEFAULT,    IMG_TRUE,         IMG_TRUE  }, /* CPU_LOCAL */
    {  PVRSRV_PHYS_HEAP_DEFAULT,    IMG_TRUE,         IMG_TRUE  }, /* GPU_PRIVATE */
    {  PVRSRV_PHYS_HEAP_GPU_LOCAL,  IMG_FALSE,        IMG_FALSE }, /* FW_MAIN */
    {  PVRSRV_PHYS_HEAP_GPU_LOCAL,  IMG_TRUE,         IMG_FALSE }, /* EXTERNAL */
    {  PVRSRV_PHYS_HEAP_GPU_LOCAL,  IMG_TRUE,         IMG_FALSE }, /* GPU_COHERENT */
    {  PVRSRV_PHYS_HEAP_GPU_LOCAL,  IMG_TRUE,         IMG_TRUE  }, /* GPU_SECURE */
    {  PVRSRV_PHYS_HEAP_FW_MAIN,    IMG_FALSE,        IMG_FALSE }, /* FW_CONFIG */
    {  PVRSRV_PHYS_HEAP_FW_MAIN,    IMG_FALSE,        IMG_FALSE }, /* FW_CODE */
    {  PVRSRV_PHYS_HEAP_FW_MAIN,    IMG_FALSE,        IMG_FALSE }, /* FW_DATA */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP0, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP0 */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP1, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP1 */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP2, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP2 */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP3, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP3 */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP4, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP4 */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP5, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP5 */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP6, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP6 */
    {  PVRSRV_PHYS_HEAP_FW_PREMAP7, IMG_FALSE,        IMG_FALSE }, /* FW_PREMAP7 */
};

2 虚拟内存管理

2.1 虚拟内存分段

IMG驱动将虚拟内存按不同用途分为若干段：

static const RGX_HEAP_INFO gasRGXHeapLayoutApp[] =
{
	/* Name                             HeapBase                                 HeapLength                               HeapReservedRegionLength                     Log2ImportAlignment pfnPresent                   HeapInstanceFlags   */
	{RGX_GENERAL_SVM_HEAP_IDENT,        RGX_GENERAL_SVM_HEAP_BASE,               RGX_GENERAL_SVM_HEAP_SIZE,               0,                                           0,                  NULL,                        HEAP_INST_DEFAULT_VALUE },
	{RGX_GENERAL_HEAP_IDENT,            RGX_GENERAL_HEAP_BASE,                   RGX_GENERAL_HEAP_SIZE,                   (1 * DEVMEM_HEAP_RESERVED_SIZE_GRANULARITY), 0,                  BRN65273IsPresent,           HEAP_INST_DEFAULT_VALUE },
	{RGX_GENERAL_HEAP_IDENT,            RGX_GENERAL_BRN_65273_HEAP_BASE,         RGX_GENERAL_BRN_65273_HEAP_SIZE,         (1 * DEVMEM_HEAP_RESERVED_SIZE_GRANULARITY), 0,                  BRN65273IsPresent,           HEAP_INST_BRN_ALT_VALUE },
	{RGX_GENERAL_NON4K_HEAP_IDENT,      RGX_GENERAL_NON4K_HEAP_BASE,             RGX_GENERAL_NON4K_HEAP_SIZE,             0,                                           0,                  BRN65273IsPresent,           HEAP_INST_DEFAULT_VALUE | HEAP_INST_NON4K_FLAG },
	{RGX_GENERAL_NON4K_HEAP_IDENT,      RGX_GENERAL_NON4K_BRN_65273_HEAP_BASE,   RGX_GENERAL_NON4K_BRN_65273_HEAP_SIZE,   0,                                           0,                  BRN65273IsPresent,           HEAP_INST_BRN_ALT_VALUE | HEAP_INST_NON4K_FLAG },
	{RGX_PDSCODEDATA_HEAP_IDENT,        RGX_PDSCODEDATA_HEAP_BASE,               RGX_PDSCODEDATA_HEAP_SIZE,               (1 * DEVMEM_HEAP_RESERVED_SIZE_GRANULARITY), 0,                  BRN65273IsPresent,           HEAP_INST_DEFAULT_VALUE },
	{RGX_PDSCODEDATA_HEAP_IDENT,        RGX_PDSCODEDATA_BRN_65273_HEAP_BASE,     RGX_PDSCODEDATA_BRN_65273_HEAP_SIZE,     (1 * DEVMEM_HEAP_RESERVED_SIZE_GRANULARITY), 0,                  BRN65273IsPresent,           HEAP_INST_BRN_ALT_VALUE },
	{RGX_RGNHDR_BRN_63142_HEAP_IDENT,   RGX_RGNHDR_BRN_63142_HEAP_BASE,          RGX_RGNHDR_BRN_63142_HEAP_SIZE,          0,                                           0,                  BRN63142IsPresent,           HEAP_INST_BRN_DEP_VALUE },
	{RGX_USCCODE_HEAP_IDENT,            RGX_USCCODE_HEAP_BASE,                   RGX_USCCODE_HEAP_SIZE,                   (1 * DEVMEM_HEAP_RESERVED_SIZE_GRANULARITY), 0,                  BRN65273IsPresent,           HEAP_INST_DEFAULT_VALUE },
	{RGX_USCCODE_HEAP_IDENT,            RGX_USCCODE_BRN_65273_HEAP_BASE,         RGX_USCCODE_BRN_65273_HEAP_SIZE,         (1 * DEVMEM_HEAP_RESERVED_SIZE_GRANULARITY), 0,                  BRN65273IsPresent,           HEAP_INST_BRN_ALT_VALUE },
	{RGX_TQ3DPARAMETERS_HEAP_IDENT,     RGX_TQ3DPARAMETERS_HEAP_BASE,            RGX_TQ3DPARAMETERS_HEAP_SIZE,            0,                                           0,                  BRN65273IsPresent,           HEAP_INST_DEFAULT_VALUE },
	{RGX_TQ3DPARAMETERS_HEAP_IDENT,     RGX_TQ3DPARAMETERS_BRN_65273_HEAP_BASE,  RGX_TQ3DPARAMETERS_BRN_65273_HEAP_SIZE,  0,                                           0,                  BRN65273IsPresent,           HEAP_INST_BRN_ALT_VALUE },
	{RGX_VK_CAPT_REPLAY_HEAP_IDENT,     RGX_VK_CAPT_REPLAY_HEAP_BASE,            RGX_VK_CAPT_REPLAY_HEAP_SIZE,            0,                                           0,                  NULL,                        HEAP_INST_DEFAULT_VALUE },
	{RGX_SIGNALS_HEAP_IDENT,            RGX_SIGNALS_HEAP_BASE,                   RGX_SIGNALS_HEAP_SIZE,                   0,                                           0,                  SignalSnoopingIsPresent,     HEAP_INST_FEAT_DEP_VALUE},
	{RGX_FBCDC_HEAP_IDENT,              RGX_FBCDC_HEAP_BASE,                     RGX_FBCDC_HEAP_SIZE,                     0,                                           0,                  FBCDescriptorIsPresent,      HEAP_INST_FEAT_DEP_VALUE},
	{RGX_FBCDC_LARGE_HEAP_IDENT,        RGX_FBCDC_LARGE_HEAP_BASE,               RGX_FBCDC_LARGE_HEAP_SIZE,               0,                                           0,                  FBCLargeDescriptorIsPresent, HEAP_INST_FEAT_DEP_VALUE},
	{RGX_CMP_MISSION_RMW_HEAP_IDENT,    RGX_CMP_MISSION_RMW_HEAP_BASE,           RGX_CMP_MISSION_RMW_HEAP_SIZE,           0,                                           0,                  NULL,                        HEAP_INST_DEFAULT_VALUE },
	{RGX_CMP_SAFETY_RMW_HEAP_IDENT,     RGX_CMP_SAFETY_RMW_HEAP_BASE,            RGX_CMP_SAFETY_RMW_HEAP_SIZE,            0,                                           0,                  NULL,                        HEAP_INST_DEFAULT_VALUE },
	{RGX_TEXTURE_STATE_HEAP_IDENT,      RGX_TEXTURE_STATE_HEAP_BASE,             RGX_TEXTURE_STATE_HEAP_SIZE,             0,                                           0,                  TextureStateIsPresent,       HEAP_INST_FEAT_DEP_VALUE},
	{RGX_VISIBILITY_TEST_HEAP_IDENT,    RGX_VISIBILITY_TEST_HEAP_BASE,           RGX_VISIBILITY_TEST_HEAP_SIZE,           0,                                           0,                  BRN65273IsPresent,           HEAP_INST_DEFAULT_VALUE },
	{RGX_VISIBILITY_TEST_HEAP_IDENT,    RGX_VISIBILITY_TEST_BRN_65273_HEAP_BASE, RGX_VISIBILITY_TEST_BRN_65273_HEAP_SIZE, 0,                                           0,                  BRN65273IsPresent,           HEAP_INST_BRN_ALT_VALUE },
	{RGX_MMU_INIA_BRN_65273_HEAP_IDENT, RGX_MMU_INIA_BRN_65273_HEAP_BASE,        RGX_MMU_INIA_BRN_65273_HEAP_SIZE,        0,                                           0,                  BRN65273IsPresent,           HEAP_INST_BRN_DEP_VALUE },
	{RGX_MMU_INIB_BRN_65273_HEAP_IDENT, RGX_MMU_INIB_BRN_65273_HEAP_BASE,        RGX_MMU_INIB_BRN_65273_HEAP_SIZE,        0,                                           0,                  BRN65273IsPresent,           HEAP_INST_BRN_DEP_VALUE }
};

static const RGX_HEAP_INFO gasRGXHeapLayoutFW[] =
{
	/* Name                          HeapBase                             HeapLength                                 HeapReservedRegionLength Log2ImportAlignment pfnIsHeapPresent     HeapInstanceFlags*/
	{RGX_FIRMWARE_MAIN_HEAP_IDENT,   RGX_FIRMWARE_MAIN_HEAP_BASE,    RGX_FIRMWARE_DEFAULT_MAIN_HEAP_SIZE,            0,                       0,                  FWBRN65101IsPresent, HEAP_INST_DEFAULT_VALUE},
	{RGX_FIRMWARE_MAIN_HEAP_IDENT,   RGX_FIRMWARE_MAIN_HEAP_BASE,    RGX_FIRMWARE_HOST_MIPS_MAIN_HEAP_SIZE_NORMAL,   0,                       0,                  FWBRN65101IsPresent, HEAP_INST_DEFAULT_VALUE},
	{RGX_FIRMWARE_MAIN_HEAP_IDENT,   RGX_FIRMWARE_MAIN_HEAP_BASE,    RGX_FIRMWARE_HOST_MIPS_MAIN_HEAP_SIZE_BRN65101, 0,                       0,                  FWBRN65101IsPresent, HEAP_INST_BRN_ALT_VALUE},
	{RGX_FIRMWARE_CONFIG_HEAP_IDENT, RGX_FIRMWARE_CONFIG_HEAP_BASE,  RGX_FIRMWARE_CONFIG_HEAP_SIZE,                  0,                       0,                  FWVZConfigPresent,   HEAP_INST_DEFAULT_VALUE},
};

gasRGXHeapLayoutApp由内核态定义分区，用户态构建相关DEVMEM_HEAP；RGX_HEAP_INFO是固件虚拟地址分段，由内核态定义和构建相关DEVMEM_HEAP。

2.2 DEVMEM_HEAP

DEVMEM_HEAP就是虚拟分段heap，定义如下：

struct DEVMEM_HEAP_TAG
{
	/* Name of heap - for debug and lookup purposes. */
	IMG_CHAR *pszName;

    # 略略略... ...
    
	IMG_DEV_VIRTADDR sBaseAddress;
	DEVMEM_SIZE_T uiSize;

    # 略略略... ...
    
	RA_ARENA *psSubAllocRA;
	IMG_CHAR *pszSubAllocRAName;

    # 略略略... ...
	RA_ARENA *psQuantizedVMRA;
	IMG_CHAR *pszQuantizedVMRAName;

    # 略略略... ...

	/* This heap is fully allocated and premapped into the device address space.
	 * Used in virtualisation for firmware heaps of Guest and optionally Host drivers. */
	IMG_BOOL bPremapped;
};

typedef struct DEVMEM_HEAP_TAG DEVMEM_HEAP;

最核心的字段包括：

虚拟地址范围：sBaseAddress和uiSize
虚拟地址管理：psQuantizedVMRA
对外内存分配：psSubAllocRA，内部实现依赖于MMU上下文和psQuantizedVMRA

各个管理器依赖如下：
请添加图片描述
注：PhysHeap和MMU CTX封装在psSubAllocRA内部实现函数中

内核态struct _PVRSRV_RGXDEV_INFO_定义了三类DEVMEM_HEAP：

typedef struct _PVRSRV_RGXDEV_INFO_
{
    # 略略略... ...
    
	/* Firmware memory context info */
	DEVMEM_CONTEXT			*psKernelDevmemCtx;
	DEVMEM_HEAP				*psFirmwareMainHeap;
	DEVMEM_HEAP				*psFirmwareConfigHeap;
	MMU_CONTEXT				*psKernelMMUCtx;
	
    # 略略略... ...
    
	/* Additional guest firmware memory context info */
	DEVMEM_HEAP				*psGuestFirmwareRawHeap[RGX_NUM_OS_SUPPORTED];
	DEVMEM_MEMDESC			*psGuestFirmwareRawMemDesc[RGX_NUM_OS_SUPPORTED];

    # 略略略... ...
    
}

psFirmwareConfigHeap是固件启动初始数据虚拟地址段；
psFirmwareMainHeap是除psFirmwareConfigHeap外的所有固件虚拟地址空间；
psGuestFirmwareRawHeap是为每个guest映射其固件内存的地址空间；

2.3 DEVMEM_CONTEXT

DEVMEM_CONTEXT是MMU CTX的封装，依赖物理heap分配页表物理页，并建立相关映射。固件（包括FwMain和FwConfig）有独立的MMU上下文，此外用户态每个渲染流程会创建MMU上下文。MMU上下文构建是通过MMU_ContextCreate函数。

3 具体实现

无论是物理heap还是设备内存heap，还是MMU。由于都是对内存区间的分配、释放、管理，其基本功能一致性，所以IMG都采用一个统一的方式去实现内存区间的管理。具体由RA系列函数和数据结构实现。
RA内存管理核心结构体是struct RA_ARENA，其中最主要的字段为pImportAlloc和per_flags_buckets：

struct _RA_ARENA_
{
	/* arena name for diagnostics output */
	IMG_CHAR name[RA_MAX_NAME_LENGTH];

    # 略略略... ...

	/* import interface, if provided */
	PFN_RA_ALLOC pImportAlloc;

    # 略略略... ...

	IMG_PSPLAY_TREE per_flags_buckets;

    # 略略略... ...

};

内存分配过程如下：

首先查看per_flags_buckets中是否有空闲的内存可分配，有则分配；否则，下一步；
调用pImportAlloc函数分配一块大内存，然后分配完将剩余的内存插入到per_flags_buckets中；

注：可以理解为per_flags_buckets是一个内存池，当内存不够的时候通过pImportAlloc扩展池的大小。
per_flags_buckets通过二叉树组织不同分配用途的内存节点(即uiFlags)，对于每类分配flag采用类似伙伴系统的方式，按2的幂次方管理空闲区间；同时在每次分配过程中，将最近使用的树节点翻转到根。
在应用中，分为不带pImportAlloc和带pImportAlloc的两种RA。前者是固定大小的RA，例如物理heap和psQuantizedVMRA；后者的pImportAlloc实现一般依赖其他RA去分配内存，例如psSubAllocRA

Victor Fung

关注

0
点赞
踩
1

收藏

觉得还不错? 一键收藏
0
评论
IMG-GPU之内存管理

IMG-GPU之内存管理引言1 物理内存管理1.1 物理内存划分1.2 原始heap1.2.1 heap配置1.2.2 heap构建1.3 物理heap备份2 虚拟内存管理3 具体实现引言IMG-GPU对划分为两个层次，底层负责具体的物理内存的管理；基于物理内存管理层构建上层的不同用户的应用内存，或者说虚拟内存管理。1 物理内存管理1.1 物理内存划分在struct _PVRSRV_DEVICE_NODE_中定义了不同的物理内存heap：typedef struct _PVRSRV_DEVICE
复制链接

扫一扫

专栏目录