DRM Memory Management

一. DRM框架中包括两种内存管理方案——TTM/GEM

Modern Linux systems require large amount of graphics memory to store frame buffers, textures, vertices and other graphics-related data. Given the very dynamic nature of many of that data, managing graphics memory efficiently is thus crucial for the graphics stack and plays a central role in the DRM infrastructure.

The DRM core includes two memory managers, namely Translation Table Manager (TTM) and Graphics Execution Manager (GEM). TTM was the first DRM memory manager to be developed and tried to be a one-size-fits-them all solution. It provides a single userspace API to accommodate the need of all hardware, supporting both Unified Memory Architecture (UMA) devices and devices with dedicated video RAM (i.e. most discrete video cards). This resulted in a large, complex piece of code that turned out to be hard to use for driver development.

GEM started as an Intel-sponsored project in reaction to TTM’s complexity. Its design philosophy is completely different: instead of providing a solution to every graphics memory-related problems, GEM identified common code between drivers and created a support library to share it. GEM has simpler initialization and execution requirements than TTM, but has no video RAM management capabilities and is thus limited to UMA devices.

一句话总结二者差异：TTM既支持集成显现，也支持独立显卡的显存管理；GEM只能支持集成显卡的显存管理。

二.TTM设计思想

The basic idea is that resources are grouped together in buffer objects of certain size and TTM handles lifetime, movement and CPU mappings of those objects.

所以，TTM主要包含两方面的工作：1.将物理memory资源抽象成确定大小buffer objects进行管理；2.管理buffer objects的生存周期/移动/CPU映射。

三.TTM实现细节

1. 如何控制CPU对buffer object memory访问的cache行为

 2.TTM如何去管理设备，比如PCIe插槽上的多张dGPU

（1）struct ttm_global

/* 全局结构，主机上所有的GPU卡共享的数据结构 */
struct ttm_global {
  struct page *dummy_read_page;
  struct list_head device_list;  // struct ttm_device链表
  atomic_t bo_count;    // 所有ttm device(GPU卡)申请的BO的计数值
};

(2) struct ttm_device

struct ttm_device {
  struct list_head device_list;  //用于将struct ttm_device挂接到ttm_global->device_list
  struct ttm_device_funcs *funcs;
  struct ttm_resource_manager sysman;
  struct ttm_resource_manager *man_drv[TTM_NUM_MEM_TYPES];
  struct drm_vma_offset_manager *vma_manager;
  struct ttm_pool pool;
  spinlock_t lru_lock;
  struct list_head ddestroy;
  struct list_head pinned;
  struct address_space *dev_mapping;
  struct delayed_work wq;
};

(3) 如何初始化一个struct ttm_device

int ttm_device_init(struct ttm_device *bdev, struct ttm_device_funcs *funcs, struct device *dev, struct address_space *mapping, struct drm_vma_offset_manager *vma_manager, bool use_dma_alloc, bool use_dma32)¶

3.TTM如何管理BO的位置以及BO位置迁移——这里的位置实际上就是物理内存资源位置

（1）struct ttm_place

Structure indicating a possible place to put an object.——表示BO的位置

struct ttm_place {
  unsigned fpfn;
  unsigned lpfn;
  uint32_t mem_type;    //One of TTM_PL_* where the resource should be allocated from
  uint32_t flags; //memory domain and caching flags for the object(这里object翻译成HBM区域比较好)
};

(2) struct ttm_placement

Structure indicating the placement you request for an object.——对某个BO进行位置迁移

struct ttm_placement {
  unsigned num_placement;
  const struct ttm_place  *placement;
  unsigned num_busy_placement;
  const struct ttm_place  *busy_placement;
};

4.TTM如何识别和管理一个dGPU上的memory types

(1) struct ttm_resource_manager

This structure is used to identify and manage memory types for a device。每种mem type对应一个该结构。

struct ttm_resource_manager {
  bool use_type;     //The memory type is enabled
  bool use_tt;
  struct ttm_device *bdev; //ttm device this manager belongs to
  uint64_t size;    //Size of the managed region.
  const struct ttm_resource_manager_func *func; //structure pointer implementing the range manager
  spinlock_t move_lock;    //lock for move fence
  struct dma_fence *move;  //The fence of the last pipelined move operation.
  struct list_head lru[TTM_MAX_BO_PRIORITY]; //The lru list for this memory type.
  uint64_t usage;    //How much of the resources are used, protected by the bdev->lru_lock.
};

(2) struct ttm_bus_placement

Structure indicating the bus placement of an object——表明bo放在哪个总线地址(VA)上

struct ttm_bus_placement {
  void *addr;  //mapped virtual address
  phys_addr_t offset; //physical addr
  bool is_iomem; //is this io memory ?
  enum ttm_caching        caching;
};

(3) struct ttm_resource

Structure indicating the placement and space resources used by a buffer object.

struct ttm_resource {
  unsigned long start;    //Start of the allocation.
  unsigned long num_pages;    //Actual size of resource in pages.
  uint32_t mem_type;    //Resource type of the allocation.
  uint32_t placement;    //Placement flags.
  struct ttm_bus_placement bus; //Placement on io bus accessible to the CPU
  struct ttm_buffer_object *bo; //weak reference to the BO, protected by ttm_device::lru_lock
  struct list_head lru;
};