DPDK学习记录13 - 内存申请1之malloc_heap_alloc

1 malloc_heap_alloc

内存初始化经过前面几步已经结束，后续就是如何去申请和管理内存。值得一提的是，由于采用了动态内存的默认配置，所以malloc heaps中的链表是空的，且memsegs list中的memseg也是按照最大limit个数初始化的，后续内存申请的过程，会动态从hugepage中申请memseg到malloc heaps中。

malloc_heap_alloc是dpdk申请内存最基本的入口，rte_malloc / rte_calloc / rte_memzone_reserve等申请内存的接口，都会调用到它。

我们来看看malloc_heap_alloc申请内存的过程。

2 malloc_heap_alloc_on_heap_id

malloc_heap_alloc调用到了malloc_heap_alloc_on_heap_id，然后再调用heap_alloc，由于malloc heaps第一次调用的时候，链表为空，所以返回值为0。尝试用alloc_more_mem_on_socket向hugepage申请更多的内存导入到malloc heaps中，然后再继续heap_alloc。

3 alloc_more_mem_on_socket

alloc_more_mem_on_socket 根据不同的page sz去调用 try_expand_heap，如果是primary进程，try_expand_heap调用try_expand_heap_primary。

3.1 try_expand_heap_primary

try_expand_heap_primary则主要调用alloc_pages_on_heap。

static int
try_expand_heap_primary(struct malloc_heap *heap, uint64_t pg_sz,
		size_t elt_size, int socket, unsigned int flags, size_t align,
		size_t bound, bool contig)
{

	alloc_sz = RTE_ALIGN_CEIL(align + elt_size +
			MALLOC_ELEM_TRAILER_LEN, pg_sz);
	n_segs = alloc_sz / pg_sz;
	ms = malloc(sizeof(*ms) * n_segs);
	memset(ms, 0, sizeof(*ms) * n_segs);
	elem = alloc_pages_on_heap(heap, pg_sz, elt_size, socket, flags, align,
			bound, contig, ms, n_segs);

	map_addr = ms[0]->addr;
	/* notify user about changes in memory map */
	eal_memalloc_mem_event_notify(RTE_MEM_EVENT_ALLOC, map_addr, alloc_sz);
	/* notify other processes that this has happened */
	if (request_sync()) {
		callback_triggered = true;
		goto free_elem;
	}
	heap->total_size += alloc_sz;
	free(ms);
	return 0;

}

3.2 alloc_pages_on_heap

alloc_pages_on_heap通过eal_memalloc_alloc_seg_bulk申请memseg，然后通过malloc_heap_add_memory加入到malloc heaps中。

struct malloc_elem *
alloc_pages_on_heap(struct malloc_heap *heap, uint64_t pg_sz, size_t elt_size,
		int socket, unsigned int flags, size_t align, size_t bound,
		bool contig, struct rte_memseg **ms, int n_segs)
{
	alloc_sz = (size_t)pg_sz * n_segs;
	allocd_pages = eal_memalloc_alloc_seg_bulk(ms, n_segs, pg_sz, socket, true);
	/* make sure we've allocated our pages... */
	if (allocd_pages < 0)
		return NULL;
	map_addr = ms[0]->addr;
	msl = rte_mem_virt2memseg_list(map_addr);

	/* add newly minted memsegs to malloc heap */
	elem = malloc_heap_add_memory(heap, msl, map_addr, alloc_sz);

	/* try once more, as now we have allocated new memory */
	ret = find_suitable_element(heap, elt_size, flags, align, bound,
			contig);

	if (ret == NULL)
		goto fail;

	return elem;

3.3 eal_memalloc_alloc_seg_bulk -> alloc_seg_walk -> alloc_seg

在hugepage的mount文件夹下多了一个rtemap_0的文件。

把rte_config->mem_config->memsegs[0]中的fbarray 对应的element设置成used，同时count置成1。

3.4 malloc_heap_add_memory

static struct malloc_elem *
malloc_heap_add_memory(struct malloc_heap *heap, struct rte_memseg_list *msl,
		void *start, size_t len)
{
	struct malloc_elem *elem = start;
	malloc_elem_init(elem, heap, msl, len);
	malloc_elem_insert(elem);
	elem = malloc_elem_join_adjacent_free(elem);
	malloc_elem_free_list_insert(elem);
	return elem;
}

此时，malloc heaps里还是空的。

malloc_heap_add_memory运行之后，heaps被写入，如下：

4 heap_alloc

struct malloc_elem *
malloc_elem_alloc(struct malloc_elem *elem, size_t size, unsigned align,
		size_t bound, bool contig)
{
	struct malloc_elem *new_elem = elem_start_pt(elem, size, align, bound,
			contig);
	const size_t old_elem_size = (uintptr_t)new_elem - (uintptr_t)elem;
	const size_t trailer_size = elem->size - old_elem_size - size -
		MALLOC_ELEM_OVERHEAD;

	malloc_elem_free_list_remove(elem);

	split_elem(elem, new_elem);
	new_elem->state = ELEM_BUSY;
	malloc_elem_free_list_insert(elem);

	return new_elem;
}

5 总结

malloc_heap_alloc是从malloc heaps中申请内存，如果当前heaps中没有找到，会尝试从hugepage中获取more memseg，然后add到heap memory中。当heaps中有足够的空间时，调用heap_alloc可以找到适当的malloc element，从该malloc element中得到一定大小的内存，剩余部分重新插入heaps中的free 链。