1、内存管理架构
2、小块空间的内存池
3、循环引用的垃圾收集
4、python中的垃圾收集
1、内存管理架构
Python内存管理机制有两套实现，由编译符号PYMALLOC_DEBUG控制，当该符号被定义时，开启debug模式下的内存管理机制，这套机制在正常内存管理动作外还记录许多关于内存的信息，方便调试。
Python内存管理机制被抽象成分层设计：
[obmalloc.c]
Object-specific allocators
_____ ______ ______ ________
[ int ] [ dict ] [ list ] ... [ string ] Python core |
+3 | | |
_______________________________ | |
[ Python's object allocator ] | |
+2 | ####### Object memory ####### | |
______________________________________________________________ |
[ Python's raw memory allocator (PyMem_ API) ] |
+1 | | |
__________________________________________________________________
[ Underlying general-purpose allocator (ex: C library malloc) ]
0 | |
=========================================================================
_______________________________________________________________________
[ OS-specific Virtual Memory Manager (VMM) ]
-1 | |
__________________________________ __________________________________
[ ] [ ]
-2 | | | |
-1层、-2层是虚拟机或操作系统和物理硬盘等的级别，我们不管。
0层是操作系统提供的内存管理接口，python用的是C运行时提供的malloc接口和free接口，这一层由操作系统实现并管理，python无法干涉这一层的行为。上面三层则是由Python实现并维护。
1层时python基于0层的包装，为Python提供一层统一的 raw memory 管理接口：
[pymem.h]
PyAPI_FUNC(void *) PyMem_Malloc(size_t);
PyAPI_FUNC(void *) PyMem_Realloc(void *, size_t);
PyAPI_FUNC(void) PyMem_Free(void *);
[object.c]void *PyMem_Malloc(size_t nbytes)
{returnPyMem_MALLOC(nbytes);
}void *PyMem_Realloc(void *p, size_t nbytes)
{returnPyMem_REALLOC(p, nbytes);
}voidPyMem_Free(void *p)
{
PyMem_FREE(p);
}
对应宏实现：
[pymem.h]#define PyMem_MALLOC(n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \:malloc((n) ? (n) : 1))#define PyMem_REALLOC(p, n) ((size_t)(n) > (size_t)PY_SSIZE_T_MAX ? NULL \:realloc((p), (n) ? (n) : 1))#define PyMem_FREE free
使用宏可减少一次函数调用提高运行效率；另一方面，对于用户使用C编写python扩展模块来说，使用宏是危险的，python内存管理的宏可能会变，导致旧版与新版python产生二进制不兼容。故使用C编写Python扩展时，使用函数接口是好习惯。
1层还提供Python中类型的内存分配器：
[pymem.h]#define PyMem_New(type, n) \( ((size_t)(n)> PY_SSIZE_T_MAX / sizeof(type)) ?NULL : \
( (type*) PyMem_Malloc((n) * sizeof(type)) ) )#define PyMem_NEW(type, n) \( ((size_t)(n)> PY_SSIZE_T_MAX / sizeof(type)) ?NULL : \
( (type*) PyMem_MALLOC((n) * sizeof(type)) ) )#define PyMem_Resize(p, type, n) \( (p)= ((size_t)(n) > PY_SSIZE_T_MAX / sizeof(type)) ?NULL : \
(type*) PyMem_Realloc((p), (n) * sizeof(type)) )#define PyMem_RESIZE(p, type, n) \( (p)= ((size_t)(n) > PY_SSIZE_T_MAX / sizeof(type)) ?NULL : \
(type*) PyMem_REALLOC((p), (n) * sizeof(type)) )#define PyMem_Del PyMem_Free
#define PyMem_DEL PyMem_FREE
PyMem_Malloc需要提供所需申请空间的大小，而PyMem_New只需提供类型和数量。
1层提供的功能是有限的，故需要2层；2层提供创建Python对象的接口(Pymalloc机制)，gc管理就在其中。
3层则是Python的一些常用对象，如整数对象，字符串对象等。
2、小块空间的内存池
对于Python中小块内存管理(不为创建对象而申请)，Python2.5中启用内存池机制，通过PyObject_Malloc、PyObject_Realloc、PyObject_Free提供。小块内存内存池也可视为一个层次结构，下到上分为：block、pool、arena和内存池。
2.1、Block
最底层有一个确定大小的内存块block。不同的block有不同的内存大小(size class)，并且是8字节对齐：
[obmalloc.c]#define ALIGNMENT 8 /* must be 2^N */
#define ALIGNMENT_SHIFT 3
#define ALIGNMENT_MASK (ALIGNMENT - 1)
block大小上限为256，申请内存超过时就使用PyMem函数族处理：
[obmalloc.h]#define SMALL_REQUEST_THRESHOLD 256
#define NB_SMALL_SIZE_CLASSES (SMALL_REQUEST_THRESHOLD / ALIGNMENT)
根据以上设定可得：
[obmalloc.c]
* Request in bytes Size of allocated block Size class idx
* ----------------------------------------------------------------
* 1-8 8 0
* 9-16 16 1
* 17-24 24 2
* 25-32 32 3
* 33-40 40 4
* 41-48 48 5
* 49-56 56 6
* 57-64 64 7
* 65-72 72 8
* ... ... ...
* 241-248 248 30
* 249-256 256 31
*
* 0, 257 and up: routed to the underlying allocator.
size class index 到 size class 的转换：
[obmalloc.c]/*Return the number of bytes in size class I, as a uint.*/
#define INDEX2SIZE(I) (((uint)(I) + 1) << ALIGNMENT_SHIFT)
block只是一个概念，在Python源码中并无实体。而管理block的则是pool。
2.2、pool
pool管理着一堆固定大小的block块，是block的集合。pool的大小通常为系统内存页(4K)：
[obmalloc.c]#define SYSTEM_PAGE_SIZE (4 * 1024)
#define SYSTEM_PAGE_SIZE_MASK (SYSTEM_PAGE_SIZE - 1)
#define POOL_SIZE SYSTEM_PAGE_SIZE /* must be 2^N */
#define POOL_SIZE_MASK SYSTEM_PAGE_SIZE_MASK
python中pool相关的结构：
[obmalloc.h]/*When you say memory, my mind reasons in terms of (pointers to) blocks*/typedef uchar block;/*Pool for small blocks.*/
structpool_header {
union { block*_padding;uint count; } ref; /*number of allocated blocks*/block*freeblock; /*pool's free list head*/
struct pool_header *nextpool; /*next pool of this size class*/
struct pool_header *prevpool; /*previous pool ""*/
uint arenaindex; /*index into arenas of base adr*/
uint szidx; /*block size class index*/
uint nextoffset; /*bytes to virgin block*/
uint maxnextoffset; /*largest valid nextoffset*/};
这是一个pool的头部，4KB除去这头部剩下的就是pool管理的内存了。
一个pool管理着一堆同样大小的block，由szidx(size class index)决定。
将4KB内存改造成pool：
申请block：
if (pool != pool->nextpool) {/** There is a used pool for this size class.
* Pick up the head block of its free list.*/
++pool->ref.count;
bp= pool->freeblock;
assert(bp!=NULL);if ((pool->freeblock = *(block **)bp) !=NULL) {
UNLOCK();return (void *)bp;
}/** Reached the end of the free list, try to extend it.*/
if (pool->nextoffset <= pool->maxnextoffset) {/*There is room for another block.*/pool->freeblock = (block*)pool +pool->nextoffset;
pool->nextoffset +=INDEX2SIZE(size);*(block **)(pool->freeblock) =NULL;
UNLOCK();return (void *)bp;
}/*Pool is full, unlink from used pools.*/next= pool->nextpool;
pool= pool->prevpool;
next->prevpool =pool;
pool->nextpool =next;
UNLOCK();return (void *)bp;
}
View Code
释放block：
voidPyObject_Free(void *p)
{
poolp pool;
block*lastfree;
poolp next, prev;uintsize;
#ifndef Py_USING_MEMORY_DEBUGGERuintarenaindex_temp;#endif
if (p == NULL) /*free(NULL) has no effect*/
return;
#ifdef WITH_VALGRINDif (UNLIKELY(running_on_valgrind > 0))gotoredirect;#endifpool=POOL_ADDR(p);if(Py_ADDRESS_IN_RANGE(p, pool)) {/*We allocated this address.*/LOCK();/*Link p to the start of the pool's freeblock list. Since
* the pool had at least the p block outstanding, the pool
* wasn't empty (so it's already in a usedpools[] list, or
* was full and is in no list -- it's not in the freeblocks
* list in any case).*/assert(pool->ref.count > 0); /*else it was empty*/
*(block **)p = lastfree = pool->freeblock;
pool->freeblock = (block *)p;if(lastfree) {struct arena_object*ao;uint nf; /*ao->nfreepools*/
/*freeblock wasn't NULL, so the pool wasn't full,
* and the pool is in a usedpools[] list.*/
if (--pool->ref.count != 0) {/*pool isn't empty: leave it in usedpools*/UNLOCK();return;
}/*Pool is now empty: unlink from usedpools, and
* link to the front of freepools. This ensures that
* previously freed pools will be allocated later
* (being not referenced, they are perhaps paged out).*/next= pool->nextpool;
prev= pool->prevpool;
next->prevpool =prev;
prev->nextpool =next;/*Link the pool to freepools. This is a singly-linked
* list, and pool->prevpool isn't used there.*/ao= &arenas[pool->arenaindex];
pool->nextpool = ao->freepools;
ao->freepools =pool;
nf= ++ao->nfreepools;/*All the rest is arena management. We just freed
* a pool, and there are 4 cases for arena mgmt:
* 1. If all the pools are free, return the arena to
* the system free().
* 2. If this is the only free pool in the arena,
* add the arena back to the `usable_arenas` list.
* 3. If the "next" arena has a smaller count of free
* pools, we have to "slide this arena right" to
* restore that usable_arenas is sorted in order of
* nfreepools.
* 4. Else there's nothing more to do.*/
if (nf == ao->ntotalpools) {/*Case 1. First unlink ao from usable_arenas.*/assert(ao->prevarena == NULL ||ao->prevarena->address != 0);
assert(ao->nextarena == NULL ||ao->nextarena->address != 0);/*Fix the pointer in the prevarena, or the
* usable_arenas pointer.*/
if (ao->prevarena ==NULL) {
usable_arenas= ao->nextarena;
assert(usable_arenas== NULL ||usable_arenas->address != 0);
}else{
assert(ao->prevarena->nextarena ==ao);
ao->prevarena->nextarena =ao->nextarena;
}/*Fix the pointer in the nextarena.*/
if (ao->nextarena !=NULL) {
assert(ao->nextarena->prevarena ==ao);
ao->nextarena->prevarena =ao->prevarena;
}/*Record that this arena_object slot is
* available to be reused.*/ao->nextarena =unused_arena_objects;
unused_arena_objects=ao;/*Free the entire arena.*/
free((void *)ao->address);
ao->address = 0; /*mark unassociated*/
--narenas_currently_allocated;
UNLOCK();return;
}if (nf == 1) {/*Case 2. Put ao at the head of
* usable_arenas. Note that because
* ao->nfreepools was 0 before, ao isn't
* currently on the usable_arenas list.*/ao->nextarena =usable_arenas;
ao->prevarena =NULL;if(usable_arenas)
usable_arenas->prevarena =ao;
usable_arenas=ao;
assert(usable_arenas->address != 0);
UNLOCK();return;
}/*If this arena is now out of order, we need to keep
* the list sorted. The list is kept sorted so that
* the "most full" arenas are used first, which allows
* the nearly empty arenas to be completely freed. In
* a few un-scientific tests, it seems like this
* approach allowed a lot more memory to be freed.*/
if (ao->nextarena == NULL ||nf<= ao->nextarena->nfreepools) {/*Case 4. Nothing to do.*/UNLOCK();return;
}/*Case 3: We have to move the arena towards the end
* of the list, because it has more free pools than
* the arena to its right.
* First unlink ao from usable_arenas.*/
if (ao->prevarena !=NULL) {/*ao isn't at the head of the list*/assert(ao->prevarena->nextarena ==ao);
ao->prevarena->nextarena = ao->nextarena;
}else{/*ao is at the head of the list*/assert(usable_arenas==ao);
usable_arenas= ao->nextarena;
}
ao->nextarena->prevarena = ao->prevarena;/*Locate the new insertion point by iterating over
* the list, using our nextarena pointer.*/
while (ao->nextarena != NULL &&nf> ao->nextarena->nfreepools) {
ao->prevarena = ao->nextarena;
ao->nextarena = ao->nextarena->nextarena;
}/*Insert ao at this point.*/assert(ao->nextarena == NULL ||ao->prevarena == ao->nextarena->prevarena);
assert(ao->prevarena->nextarena == ao->nextarena);
ao->prevarena->nextarena =ao;if (ao->nextarena !=NULL)
ao->nextarena->prevarena =ao;/*Verify that the swaps worked.*/assert(ao->nextarena == NULL ||nf<= ao->nextarena->nfreepools);
assert(ao->prevarena == NULL ||nf> ao->prevarena->nfreepools);
assert(ao->nextarena == NULL ||ao->nextarena->prevarena ==ao);
assert((usable_arenas== ao &&ao->prevarena == NULL) ||ao->prevarena->nextarena ==ao);
UNLOCK();return;
}/*Pool was full, so doesn't currently live in any list:
* link it to the front of the appropriate usedpools[] list.
* This mimics LRU pool usage for new allocations and
* targets optimal filling when several pools contain
* blocks of the same size class.*/
--pool->ref.count;
assert(pool->ref.count > 0); /*else the pool is empty*/size= pool->szidx;
next= usedpools[size +size];
prev= next->prevpool;/*insert pool before next: prev pool next*/pool->nextpool =next;
pool->prevpool =prev;
next->prevpool =pool;
prev->nextpool =pool;
UNLOCK();return;
}
#ifdef WITH_VALGRIND
redirect:#endif
/*We didn't allocate this address.*/
free(p);
}
View Code
释放后freeblock会调整指到释放了的blobk上，有效利用空闲block。
block分配的一般行为：
[obmalloc.c]-[allocate block]
...if (pool != pool->nextpool) {/** There is a used pool for this size class.
* Pick up the head block of its free list.*/
++pool->ref.count;
bp= pool->freeblock;
assert(bp!=NULL);if ((pool->freeblock = *(block **)bp) !=NULL) {
UNLOCK();return (void *)bp;
}
...if (pool->nextoffset <= pool->maxnextoffset) {
...
}
...
}
freeblock为空证明pool满了，会提供另一个pool。而pool的集合则是arena。
2.3、arena
arena是多个pool的聚合。Pyhton中arena的默认大小为256KB(可装64个pool)：
[obmalloc.c]#define ARENA_SIZE (256 << 10) /* 256KB */
Python中的arena：
[obmalloc.c]
typedef uchar block;/*Record keeping for arenas.*/
structarena_object {/*The address of the arena, as returned by malloc. Note that 0
* will never be returned by a successful malloc, and is used
* here to mark an arena_object that doesn't correspond to an
* allocated arena.*/uptr address;/*Pool-aligned pointer to the next pool to be carved off.*/block*pool_address;/*The number of available pools in the arena: free pools + never-
* allocated pools.*/
uintnfreepools;/*The total number of pools in the arena, whether or not available.*/
uintntotalpools;/*Singly-linked list of available pools.*/
struct pool_header*freepools;/*Whenever this arena_object is not associated with an allocated
* arena, the nextarena member is used to link all unassociated
* arena_objects in the singly-linked `unused_arena_objects` list.
* The prevarena member is unused in this case.
*
* When this arena_object is associated with an allocated arena
* with at least one available pool, both members are used in the
* doubly-linked `usable_arenas` list, which is maintained in
* increasing order of `nfreepools` values.
*
* Else this arena_object is associated with an allocated arena
* all of whose pools are in use. `nextarena` and `prevarena`
* are both meaningless in this case.*/
struct arena_object*nextarena;struct arena_object*prevarena;
};
一个完整的arena是 一个arena_object和其管理的pool集合；
一个完整的pool时一个 pool_header 和其管理的block集合。
pool_header和其管理的block内存上是连续的，而arena则是分离：
差别体现在申请pool_header时其所管理的内存被申请了，而arena_object则没有。
当一个arena_object没与pool集合建立联系时，arena处于“未使用”状态，否则进入“可用”状态。未使用的单向连接(unused_arena_objects)，可用的双向连接(usable_arenas)。
arena的申请new_arena：
[obmalloc.c]/*Array of objects used to track chunks of memory (arenas).*/
static struct arena_object* arenas =NULL;/*Number of slots currently allocated in the `arenas` vector.*/
static uint maxarenas = 0;/*The head of the singly-linked, NULL-terminated list of available
* arena_objects.*/
static struct arena_object* unused_arena_objects =NULL;/*The head of the doubly-linked, NULL-terminated at each end, list of
* arena_objects associated with arenas that have pools available.*/
static struct arena_object* usable_arenas =NULL;/*How many arena_objects do we initially allocate?
* 16 = can allocate 16 arenas = 16 * ARENA_SIZE = 4MB before growing the
* `arenas` vector.*/
#define INITIAL_ARENA_OBJECTS 16
/*Number of arenas allocated that haven't been free()'d.*/
static size_t narenas_currently_allocated = 0;
#ifdef PYMALLOC_DEBUG/*Total number of times malloc() called to allocate an arena.*/
static size_t ntimes_arena_allocated = 0;/*High water mark (max value ever seen) for narenas_currently_allocated.*/
static size_t narenas_highwater = 0;#endif
/*Allocate a new arena. If we run out of memory, return NULL. Else
* allocate a new arena, and return the address of an arena_object
* describing the new arena. It's expected that the caller will set
* `usable_arenas` to the return value.*/
static struct arena_object*new_arena(void)
{struct arena_object*arenaobj;uint excess; /*number of bytes above pool alignment*/#ifdef PYMALLOC_DEBUGif (Py_GETENV("PYTHONMALLOCSTATS"))
_PyObject_DebugMallocStats();#endif
if (unused_arena_objects ==NULL) {uinti;uintnumarenas;
size_t nbytes;/*Double the number of arena objects on each allocation.
* Note that it's possible for `numarenas` to overflow.*/numarenas= maxarenas ? maxarenas << 1: INITIAL_ARENA_OBJECTS;if (numarenas <=maxarenas)return NULL; /*overflow*/
#if SIZEOF_SIZE_T <= SIZEOF_INT
if (numarenas > PY_SIZE_MAX / sizeof(*arenas))return NULL; /*overflow*/
#endifnbytes= numarenas * sizeof(*arenas);
arenaobj= (struct arena_object *)realloc(arenas, nbytes);if (arenaobj ==NULL)returnNULL;
arenas=arenaobj;/*We might need to fix pointers that were copied. However,
* new_arena only gets called when all the pages in the
* previous arenas are full. Thus, there are *no* pointers
* into the old array. Thus, we don't have to worry about
* invalid pointers. Just to be sure, some asserts:*/assert(usable_arenas==NULL);
assert(unused_arena_objects==NULL);/*Put the new arenas on the unused_arena_objects list.*/
for (i = maxarenas; i < numarenas; ++i) {
arenas[i].address= 0; /*mark as unassociated*/arenas[i].nextarena= i < numarenas - 1 ?
&arenas[i+1] : NULL;
}/*Update globals.*/unused_arena_objects= &arenas[maxarenas];
maxarenas=numarenas;
}/*Take the next available arena object off the head of the list.*/assert(unused_arena_objects!=NULL);
arenaobj=unused_arena_objects;
unused_arena_objects= arenaobj->nextarena;
assert(arenaobj->address == 0);
arenaobj->address = (uptr)malloc(ARENA_SIZE);if (arenaobj->address == 0) {/*The allocation failed: return NULL after putting the
* arenaobj back.*/arenaobj->nextarena =unused_arena_objects;
unused_arena_objects=arenaobj;returnNULL;
}++narenas_currently_allocated;
#ifdef PYMALLOC_DEBUG++ntimes_arena_allocated;if (narenas_currently_allocated >narenas_highwater)
narenas_highwater=narenas_currently_allocated;#endifarenaobj->freepools =NULL;/*pool_address
nfreepools pool_address = (block*)arenaobj->address;
arenaobj->nfreepools = ARENA_SIZE /POOL_SIZE;
assert(POOL_SIZE* arenaobj->nfreepools ==ARENA_SIZE);
excess= (uint)(arenaobj->address &POOL_SIZE_MASK);if (excess != 0) {--arenaobj->nfreepools;
arenaobj->pool_address += POOL_SIZE -excess;
}
arenaobj->ntotalpools = arenaobj->nfreepools;returnarenaobj;
}
View Code
先检查unused_arena_objects中是否有“未使用”的arena，有则从中取；否则新增arenas(并调整maxarenas的值)；
先申请ARENA_SIZE(256KB)的内存块，将其变成“可用”，然后设置一些维护pool的信息，后被usable_arenas接收；
address标记arena_object状态(“未使用”还是“可用”)。
2.4、内存池
小块内存池大小限制由SMALL_MEMORY_LIMIT控制，默认不限制：
/** Maximum amount of memory managed by the allocator for small requests.*/#ifdef WITH_MEMORY_LIMITS
#ifndef SMALL_MEMORY_LIMIT#define SMALL_MEMORY_LIMIT (64 * 1024 * 1024) /* 64 MB -- more? */
#endif
#endif
/** The allocator sub-allocates blocks of memory (called arenas) aligned
* on a page boundary. This is a reserved virtual address space for the
* current process (obtained through a malloc call). In no way this means
* that the memory arenas will be used entirely. A malloc() is usually
* an address range reservation for bytes, unless all pages within this
* space are referenced subsequently. So malloc'ing big blocks and not using
* them does not mean "wasting memory". It's an addressable range wastage...
*
* Therefore, allocating arenas with malloc is not optimal, because there is
* some address space wastage, but this is the most portable way to request
* memory from the system across various platforms.*/
#define ARENA_SIZE (256 << 10) /* 256KB */#ifdef WITH_MEMORY_LIMITS#define MAX_ARENAS (SMALL_MEMORY_LIMIT / ARENA_SIZE)
#endif
虽然arena是Python小块内存池的最上层结构，但申请内存时不与它打交道，而是直接以pool作为基本操作单元。同一个arena里面可能管理着 管理不同大小block的pool。
pool在python运行时处于used状态、full状态或empty状态中的一种。arena包含三种状态pool的集合的一个可能状态：
看下维护used状态pool的usedpools：
[obmalloc.c]
typedef uchar block;#define PTA(x) ((poolp )((uchar *)&(usedpools[2*(x)]) - 2*sizeof(block *)))
#define PT(x) PTA(x), PTA(x)
static poolp usedpools[2 * ((NB_SMALL_SIZE_CLASSES + 7) / 8) * 8] ={
PT(0), PT(1), PT(2), PT(3), PT(4), PT(5), PT(6), PT(7)#if NB_SMALL_SIZE_CLASSES > 8, PT(8), PT(9), PT(10), PT(11), PT(12), PT(13), PT(14), PT(15)#if NB_SMALL_SIZE_CLASSES > 16, PT(16), PT(17), PT(18), PT(19), PT(20), PT(21), PT(22), PT(23)#if NB_SMALL_SIZE_CLASSES > 24, PT(24), PT(25), PT(26), PT(27), PT(28), PT(29), PT(30), PT(31)#if NB_SMALL_SIZE_CLASSES > 32, PT(32), PT(33), PT(34), PT(35), PT(36), PT(37), PT(38), PT(39)#if NB_SMALL_SIZE_CLASSES > 40, PT(40), PT(41), PT(42), PT(43), PT(44), PT(45), PT(46), PT(47)#if NB_SMALL_SIZE_CLASSES > 48, PT(48), PT(49), PT(50), PT(51), PT(52), PT(53), PT(54), PT(55)#if NB_SMALL_SIZE_CLASSES > 56, PT(56), PT(57), PT(58), PT(59), PT(60), PT(61), PT(62), PT(63)#endif /* NB_SMALL_SIZE_CLASSES > 56 */
#endif /* NB_SMALL_SIZE_CLASSES > 48 */
#endif /* NB_SMALL_SIZE_CLASSES > 40 */
#endif /* NB_SMALL_SIZE_CLASSES > 32 */
#endif /* NB_SMALL_SIZE_CLASSES > 24 */
#endif /* NB_SMALL_SIZE_CLASSES > 16 */
#endif /* NB_SMALL_SIZE_CLASSES > 8 */};
其中 NB_SMALL_SIZE_CLASSES 指明一共有多少个size class：
[obmalloc.c]#define SMALL_REQUEST_THRESHOLD 256
#define NB_SMALL_SIZE_CLASSES (SMALL_REQUEST_THRESHOLD / ALIGNMENT)
Python启动后usedpools中无可用pool。Python采用延迟分配策略，当我们申请小块内存时才分配。
初始分配空间代码PyObject_Malloc：
#undef PyObject_Malloc
void *PyObject_Malloc(size_t nbytes)
{
block*bp;
poolp pool;
poolp next;uintsize;
#ifdef WITH_VALGRINDif (UNLIKELY(running_on_valgrind == -1))
running_on_valgrind=RUNNING_ON_VALGRIND;if(UNLIKELY(running_on_valgrind))gotoredirect;#endif
/** Limit ourselves to PY_SSIZE_T_MAX bytes to prevent security holes.
* Most python internals blindly use a signed Py_ssize_t to track
* things without checking for overflows or negatives.
* As size_t is unsigned, checking for nbytes < 0 is not required.*/
if (nbytes >PY_SSIZE_T_MAX)returnNULL;/** This implicitly redirects malloc(0).*/
if ((nbytes - 1)
LOCK();/** Most frequent paths first*/size= (uint)(nbytes - 1) >>ALIGNMENT_SHIFT;
pool= usedpools[size +size];if (pool != pool->nextpool) {/** There is a used pool for this size class.
* Pick up the head block of its free list.*/
++pool->ref.count;
bp= pool->freeblock;
assert(bp!=NULL);if ((pool->freeblock = *(block **)bp) !=NULL) {
UNLOCK();return (void *)bp;
}/** Reached the end of the free list, try to extend it.*/
if (pool->nextoffset <= pool->maxnextoffset) {/*There is room for another block.*/pool->freeblock = (block*)pool +pool->nextoffset;
pool->nextoffset +=INDEX2SIZE(size);*(block **)(pool->freeblock) =NULL;
UNLOCK();return (void *)bp;
}/*Pool is full, unlink from used pools.*/next= pool->nextpool;
pool= pool->prevpool;
next->prevpool =pool;
pool->nextpool =next;
UNLOCK();return (void *)bp;
}/*There isn't a pool of the right size class immediately
* available: use a free pool.*/
if (usable_arenas ==NULL) {/*No arena has a free pool: allocate a new arena.*/#ifdef WITH_MEMORY_LIMITSif (narenas_currently_allocated >=MAX_ARENAS) {
UNLOCK();gotoredirect;
}#endifusable_arenas=new_arena();if (usable_arenas ==NULL) {
UNLOCK();gotoredirect;
}
usable_arenas->nextarena =usable_arenas->prevarena =NULL;
}
assert(usable_arenas->address != 0);/*Try to get a cached free pool.*/pool= usable_arenas->freepools;if (pool !=NULL) {/*Unlink from cached pools.*/usable_arenas->freepools = pool->nextpool;/*This arena already had the smallest nfreepools
* value, so decreasing nfreepools doesn't change
* that, and we don't need to rearrange the
* usable_arenas list. However, if the arena has
* become wholly allocated, we need to remove its
* arena_object from usable_arenas.*/
--usable_arenas->nfreepools;if (usable_arenas->nfreepools == 0) {/*Wholly allocated: remove.*/assert(usable_arenas->freepools ==NULL);
assert(usable_arenas->nextarena == NULL ||usable_arenas->nextarena->prevarena ==usable_arenas);
usable_arenas= usable_arenas->nextarena;if (usable_arenas !=NULL) {
usable_arenas->prevarena =NULL;
assert(usable_arenas->address != 0);
}
}else{/*nfreepools > 0: it must be that freepools
* isn't NULL, or that we haven't yet carved
* off all the arena's pools for the first
* time.*/assert(usable_arenas->freepools != NULL ||usable_arenas->pool_address <=(block*)usable_arenas->address +ARENA_SIZE-POOL_SIZE);
}
init_pool:/*Frontlink to used pools.*/next= usedpools[size + size]; /*== prev*/pool->nextpool =next;
pool->prevpool =next;
next->nextpool =pool;
next->prevpool =pool;
pool->ref.count = 1;if (pool->szidx ==size) {/*Luckily, this pool last contained blocks
* of the same size class, so its header
* and free list are already initialized.*/bp= pool->freeblock;
pool->freeblock = *(block **)bp;
UNLOCK();return (void *)bp;
}/** Initialize the pool header, set up the free list to
* contain just the second block, and return the first
* block.*/pool->szidx =size;
size=INDEX2SIZE(size);
bp= (block *)pool +POOL_OVERHEAD;
pool->nextoffset = POOL_OVERHEAD + (size << 1);
pool->maxnextoffset = POOL_SIZE -size;
pool->freeblock = bp +size;*(block **)(pool->freeblock) =NULL;
UNLOCK();return (void *)bp;
}/*Carve off a new pool.*/assert(usable_arenas->nfreepools > 0);
assert(usable_arenas->freepools ==NULL);
pool= (poolp)usable_arenas->pool_address;
assert((block*)pool <= (block*)usable_arenas->address +ARENA_SIZE-POOL_SIZE);
pool->arenaindex = usable_arenas -arenas;
assert(&arenas[pool->arenaindex] ==usable_arenas);
pool->szidx =DUMMY_SIZE_IDX;
usable_arenas->pool_address +=POOL_SIZE;--usable_arenas->nfreepools;if (usable_arenas->nfreepools == 0) {
assert(usable_arenas->nextarena == NULL ||usable_arenas->nextarena->prevarena ==usable_arenas);/*Unlink the arena: it is completely allocated.*/usable_arenas= usable_arenas->nextarena;if (usable_arenas !=NULL) {
usable_arenas->prevarena =NULL;
assert(usable_arenas->address != 0);
}
}gotoinit_pool;
}/*The small block allocator ends here.*/redirect:/*Redirect the original request to the underlying (libc) allocator.
* We jump here on bigger requests, on error in the code above (as a
* last chance to serve the request) or when the max memory limit
* has been reached.*/
if (nbytes == 0)
nbytes= 1;return (void *)malloc(nbytes);
}
View Code
开始如果usable_arenas为空，则从new_arena申请一个arena，再构建usable_arenas链表，从usable_arenas取一个可用pool。取完后如arena无可用pool，将其移出usable_arenas。
取到pool后将其放到usedpools中，然后对pool进行初始化，返回相应block。
python2.5后，将arena内存泄漏问题修复(arena申请pool但从不释放)，回收代码PyObject_Free：
#undef PyObject_Free
voidPyObject_Free(void *p)
{
poolp pool;
block*lastfree;
poolp next, prev;uintsize;
#ifndef Py_USING_MEMORY_DEBUGGERuintarenaindex_temp;#endif
if (p == NULL) /*free(NULL) has no effect*/
return;
#ifdef WITH_VALGRINDif (UNLIKELY(running_on_valgrind > 0))gotoredirect;#endifpool=POOL_ADDR(p);if(Py_ADDRESS_IN_RANGE(p, pool)) {/*We allocated this address.*/LOCK();/*Link p to the start of the pool's freeblock list. Since
* the pool had at least the p block outstanding, the pool
* wasn't empty (so it's already in a usedpools[] list, or
* was full and is in no list -- it's not in the freeblocks
* list in any case).*/assert(pool->ref.count > 0); /*else it was empty*/
*(block **)p = lastfree = pool->freeblock;
pool->freeblock = (block *)p;if(lastfree) {struct arena_object*ao;uint nf; /*ao->nfreepools*/
/*freeblock wasn't NULL, so the pool wasn't full,
* and the pool is in a usedpools[] list.*/
if (--pool->ref.count != 0) {/*pool isn't empty: leave it in usedpools*/UNLOCK();return;
}/*Pool is now empty: unlink from usedpools, and
* link to the front of freepools. This ensures that
* previously freed pools will be allocated later
* (being not referenced, they are perhaps paged out).*/next= pool->nextpool;
prev= pool->prevpool;
next->prevpool =prev;
prev->nextpool =next;/*Link the pool to freepools. This is a singly-linked
* list, and pool->prevpool isn't used there.*/ao= &arenas[pool->arenaindex];
pool->nextpool = ao->freepools;
ao->freepools =pool;
nf= ++ao->nfreepools;/*All the rest is arena management. We just freed
* a pool, and there are 4 cases for arena mgmt:
* 1. If all the pools are free, return the arena to
* the system free().
* 2. If this is the only free pool in the arena,
* add the arena back to the `usable_arenas` list.
* 3. If the "next" arena has a smaller count of free
* pools, we have to "slide this arena right" to
* restore that usable_arenas is sorted in order of
* nfreepools.
* 4. Else there's nothing more to do.*/
if (nf == ao->ntotalpools) {/*Case 1. First unlink ao from usable_arenas.*/assert(ao->prevarena == NULL ||ao->prevarena->address != 0);
assert(ao->nextarena == NULL ||ao->nextarena->address != 0);/*Fix the pointer in the prevarena, or the
* usable_arenas pointer.*/
if (ao->prevarena ==NULL) {
usable_arenas= ao->nextarena;
assert(usable_arenas== NULL ||usable_arenas->address != 0);
}else{
assert(ao->prevarena->nextarena ==ao);
ao->prevarena->nextarena =ao->nextarena;
}/*Fix the pointer in the nextarena.*/
if (ao->nextarena !=NULL) {
assert(ao->nextarena->prevarena ==ao);
ao->nextarena->prevarena =ao->prevarena;
}/*Record that this arena_object slot is
* available to be reused.*/ao->nextarena =unused_arena_objects;
unused_arena_objects=ao;/*Free the entire arena.*/
free((void *)ao->address);
ao->address = 0; /*mark unassociated*/
--narenas_currently_allocated;
UNLOCK();return;
}if (nf == 1) {/*Case 2. Put ao at the head of
* usable_arenas. Note that because
* ao->nfreepools was 0 before, ao isn't
* currently on the usable_arenas list.*/ao->nextarena =usable_arenas;
ao->prevarena =NULL;if(usable_arenas)
usable_arenas->prevarena =ao;
usable_arenas=ao;
assert(usable_arenas->address != 0);
UNLOCK();return;
}/*If this arena is now out of order, we need to keep
* the list sorted. The list is kept sorted so that
* the "most full" arenas are used first, which allows
* the nearly empty arenas to be completely freed. In
* a few un-scientific tests, it seems like this
* approach allowed a lot more memory to be freed.*/
if (ao->nextarena == NULL ||nf<= ao->nextarena->nfreepools) {/*Case 4. Nothing to do.*/UNLOCK();return;
}/*Case 3: We have to move the arena towards the end
* of the list, because it has more free pools than
* the arena to its right.
* First unlink ao from usable_arenas.*/
if (ao->prevarena !=NULL) {/*ao isn't at the head of the list*/assert(ao->prevarena->nextarena ==ao);
ao->prevarena->nextarena = ao->nextarena;
}else{/*ao is at the head of the list*/assert(usable_arenas==ao);
usable_arenas= ao->nextarena;
}
ao->nextarena->prevarena = ao->prevarena;/*Locate the new insertion point by iterating over
* the list, using our nextarena pointer.*/
while (ao->nextarena != NULL &&nf> ao->nextarena->nfreepools) {
ao->prevarena = ao->nextarena;
ao->nextarena = ao->nextarena->nextarena;
}/*Insert ao at this point.*/assert(ao->nextarena == NULL ||ao->prevarena == ao->nextarena->prevarena);
assert(ao->prevarena->nextarena == ao->nextarena);
ao->prevarena->nextarena =ao;if (ao->nextarena !=NULL)
ao->nextarena->prevarena =ao;/*Verify that the swaps worked.*/assert(ao->nextarena == NULL ||nf<= ao->nextarena->nfreepools);
assert(ao->prevarena == NULL ||nf> ao->prevarena->nfreepools);
assert(ao->nextarena == NULL ||ao->nextarena->prevarena ==ao);
assert((usable_arenas== ao &&ao->prevarena == NULL) ||ao->prevarena->nextarena ==ao);
UNLOCK();return;
}/*Pool was full, so doesn't currently live in any list:
* link it to the front of the appropriate usedpools[] list.
* This mimics LRU pool usage for new allocations and
* targets optimal filling when several pools contain
* blocks of the same size class.*/
--pool->ref.count;
assert(pool->ref.count > 0); /*else the pool is empty*/size= pool->szidx;
next= usedpools[size +size];
prev= next->prevpool;/*insert pool before next: prev pool next*/pool->nextpool =next;
pool->prevpool =prev;
next->prevpool =pool;
prev->nextpool =pool;
UNLOCK();return;
}
#ifdef WITH_VALGRIND
redirect:#endif
/*We didn't allocate this address.*/
free(p);
}
View Code
1、如果arena中所有pool都是empty，释放pool集合所占内存；
2、如果之前arena没有empty的pool，多一个后将arena移到usable_arenas中；
3、usable_arenas时一个有序链表，nfreepools个数递增，保证一个arena empty pool个数越多被使用机会越少。从而保证多余内存被释放并归还系统；
4、其他情况不对arena进行处理；
3、循环引用的垃圾收集
python通过引用计数实时内存管理，优点是具有实时性，缺点是带来维护引用计数额外操作、更多的内存分配与释放。python设计了大量内存池，除了第2节提到的小块内存的内存池，对其他python对象也有内存池机制，以此弥补引用计数软肋。
引用计数还有一致命弱点----循环引用，python引入标记-清除以及分代回收填补此漏洞。
垃圾回收分两阶段：垃圾检测和垃圾回收。
python垃圾收集的过程：
4、python中的垃圾收集
4.1、可收集对象链表
python中循环引用发生在container对象间，用PyGC_Head变成可收集对象(进入可收集对象链表)：
[objimpl.h]/*GC information is stored BEFORE the object structure.*/typedef union _gc_head {struct{
union _gc_head*gc_next;
union _gc_head*gc_prev;
Py_ssize_t gc_refs;
} gc;long double dummy; /*force worst-case alignment*/} PyGC_Head;
container创建过程：
[gcmodule.c]
PyObject*_PyObject_GC_New(PyTypeObject*tp)
{
PyObject*op =_PyObject_GC_Malloc(_PyObject_SIZE(tp));if (op !=NULL)
op=PyObject_INIT(op, tp);returnop;
}
PyObject *_PyObject_GC_Malloc(size_t basicsize)
{
PyObject*op;
PyGC_Head*g;if (basicsize > PY_SSIZE_T_MAX - sizeof(PyGC_Head))returnPyErr_NoMemory();
g= (PyGC_Head *)PyObject_MALLOC(sizeof(PyGC_Head) +basicsize);if (g ==NULL)returnPyErr_NoMemory();
g->gc.gc_refs =GC_UNTRACKED;
generations[0].count++; /*number of allocated GC objects*/
if (generations[0].count > generations[0].threshold &&enabled&&generations[0].threshold &&
!collecting &&
!PyErr_Occurred()) {
collecting= 1;
collect_generations();
collecting= 0;
}
op=FROM_GC(g);returnop;
}
创建后第一部分是用于垃圾收集的PyGC_Head，接着是python所有对象都有的PyObject_HEAD，最后是属于container对象自身的数据。
PyGC_Head和PyObject_HEAD地址转换：
[gcmodule.c]/*Get an object's GC head*/
#define AS_GC(o) ((PyGC_Head *)(o)-1)
/*Get the object given the GC head*/
#define FROM_GC(g) ((PyObject *)(((PyGC_Head *)g)+1))[objimpl.h]#define _Py_AS_GC(o) ((PyGC_Head *)(o)-1)
在创建某个container对象最后一步会链接到可收集对象链表中：
[objimpl.h]/*Tell the GC to track this object. NB: While the object is tracked the
* collector it must be safe to call the ob_traverse method.*/
#define _PyObject_GC_TRACK(o) do { \PyGC_Head*g =_Py_AS_GC(o); \if (g->gc.gc_refs !=_PyGC_REFS_UNTRACKED) \
Py_FatalError("GC object already tracked"); \
g->gc.gc_refs =_PyGC_REFS_REACHABLE; \
g->gc.gc_next =_PyGC_generation0; \
g->gc.gc_prev = _PyGC_generation0->gc.gc_prev; \
g->gc.gc_prev->gc.gc_next =g; \
_PyGC_generation0->gc.gc_prev =g; \
}while (0);
从链表摘除container对象：
[objimpl.h]/*Tell the GC to stop tracking this object.
* gc_next doesn't need to be set to NULL, but doing so is a good
* way to provoke memory errors if calling code is confused.*/
#define _PyObject_GC_UNTRACK(o) do { \PyGC_Head*g =_Py_AS_GC(o); \
assert(g->gc.gc_refs !=_PyGC_REFS_UNTRACKED); \
g->gc.gc_refs =_PyGC_REFS_UNTRACKED; \
g->gc.gc_prev->gc.gc_next = g->gc.gc_next; \
g->gc.gc_next->gc.gc_prev = g->gc.gc_prev; \
g->gc.gc_next =NULL; \
}while (0);
4.2、分代的垃圾收集
python中引入分代的垃圾收集机制，共有3代，每一代都是一个链表，在之前的链表基础上加上一个表头：
[gcmodule.c]structgc_generation {
PyGC_Head head;int threshold; /*collection threshold*/
int count; /*count of allocations or collections of younger
generations*/};
python中维护 了三个gc_generation结构的数组，通过这数组控制三条可收集对象链表，即三个“代”：
[gcmodule.c]#define NUM_GENERATIONS 3
#define GEN_HEAD(n) (&generations[n].head)
/*linked lists of container objects*/
static struct gc_generation generations[NUM_GENERATIONS] ={/*PyGC_Head, threshold, count*/{{{GEN_HEAD(0), GEN_HEAD(0), 0}}, 700, 0},
{{{GEN_HEAD(1), GEN_HEAD(1), 0}}, 10, 0},
{{{GEN_HEAD(2), GEN_HEAD(2), 0}}, 10, 0},
};
PyGC_Head*_PyGC_generation0 = GEN_HEAD(0);
count表示有多少个可收集对象，threshold表示该链可容纳收集对象个数，当超过这个值时会触发垃圾回收机制：
[gcmodule.c]staticPy_ssize_t
collect_generations(void)
{inti;
Py_ssize_t n= 0;/*Find the oldest generation (highest numbered) where the count
* exceeds the threshold. Objects in the that generation and
* generations younger than it will be collected.*/
for (i = NUM_GENERATIONS-1; i >= 0; i--) {if (generations[i].count >generations[i].threshold) {/*Avoid quadratic performance degradation in number
of tracked objects. See comments at the beginning
of this file, and issue #4074.*/
if (i == NUM_GENERATIONS - 1
&& long_lived_pending < long_lived_total / 4)continue;
n=collect(i);break;
}
}returnn;
}
4.3、Python中的标记——清除方法
开始垃圾收集前，会将收集的代及更年轻的代合并，再进行收集：
[gcmodule.c]static voidgc_list_init(PyGC_Head*list)
{
list->gc.gc_prev =list;
list->gc.gc_next =list;
}/*append list `from` onto list `to`; `from` becomes an empty list*/
static voidgc_list_merge(PyGC_Head*from, PyGC_Head *to)
{
PyGC_Head*tail;
assert(from !=to);if (!gc_list_is_empty(from)) {
tail= to->gc.gc_prev;
tail->gc.gc_next = from->gc.gc_next;
tail->gc.gc_next->gc.gc_prev =tail;
to->gc.gc_prev = from->gc.gc_prev;
to->gc.gc_prev->gc.gc_next =to;
}
gc_list_init(from);
}
为了得出真正的引用计数，引入有效引入计数，使用计数副本计算，即PyGC_Head中的gc.gc_ref：
[gcmodule.c]static voidupdate_refs(PyGC_Head*containers)
{
PyGC_Head*gc = containers->gc.gc_next;for (; gc != containers; gc = gc->gc.gc_next) {
assert(gc->gc.gc_refs ==GC_REACHABLE);
gc->gc.gc_refs =Py_REFCNT(FROM_GC(gc));/*Python's cyclic gc should never see an incoming refcount
* of 0: if something decref'ed to 0, it should have been
* deallocated immediately at that time.
* Possible cause (if the assert triggers): a tp_dealloc
* routine left a gc-aware object tracked during its teardown
* phase, and did something-- or allowed something to happen --
* that called back into Python. gc can trigger then, and may
* see the still-tracked dying object. Before this assert
* was added, such mistakes went on to allow gc to try to
* delete the object again. In a debug build, that caused
* a mysterious segfault, when _Py_ForgetReference tried
* to remove the object from the doubly-linked list of all
* objects a second time. In a release build, an actual
* double deallocation occurred, which leads to corruption
* of the allocator's internal bookkeeping pointers. That's
* so serious that maybe this should be a release-build
* check instead of an assert?*/assert(gc->gc.gc_refs != 0);
}
}
先将对象gc.gc_ref设置为ob_refcnt的值，再将循环引用摘除：
[gcmodule.c]static voidsubtract_refs(PyGC_Head*containers)
{
traverseproc traverse;
PyGC_Head*gc = containers->gc.gc_next;for (; gc != containers; gc=gc->gc.gc_next) {
traverse= Py_TYPE(FROM_GC(gc))->tp_traverse;
(void) traverse(FROM_GC(gc),
(visitproc)visit_decref,
NULL);
}
}
traverse与特定的container对象相关，用于遍历container对象中的每一个引用，对引用作某种动作，在subtract_refs中动作就是visit_dec_ref。完成后摘除了container对象间的环引用，得出root object(用于开始标记--清除算法)集合。
得出root object集合后，开始标记垃圾，用move_unreachable将可回收对象从root object链表中移到unreachable链表中：
[gcmodule.c]static voidmove_unreachable(PyGC_Head*young, PyGC_Head *unreachable)
{
PyGC_Head*gc = young->gc.gc_next;/*Invariants: all objects "to the left" of us in young have gc_refs
* = GC_REACHABLE, and are indeed reachable (directly or indirectly)
* from outside the young list as it was at entry. All other objects
* from the original young "to the left" of us are in unreachable now,
* and have gc_refs = GC_TENTATIVELY_UNREACHABLE. All objects to the
* left of us in 'young' now have been scanned, and no objects here
* or to the right have been scanned yet.*/
while (gc !=young) {
PyGC_Head*next;if (gc->gc.gc_refs) {/*gc is definitely reachable from outside the
* original 'young'. Mark it as such, and traverse
* its pointers to find any other objects that may
* be directly reachable from it. Note that the
* call to tp_traverse may append objects to young,
* so we have to wait until it returns to determine
* the next object to visit.*/PyObject*op =FROM_GC(gc);
traverseproc traverse= Py_TYPE(op)->tp_traverse;
assert(gc->gc.gc_refs > 0);
gc->gc.gc_refs =GC_REACHABLE;
(void) traverse(op,
(visitproc)visit_reachable,
(void *)young);
next= gc->gc.gc_next;if(PyTuple_CheckExact(op)) {
_PyTuple_MaybeUntrack(op);
}
}else{/*This *may* be unreachable. To make progress,
* assume it is. gc isn't directly reachable from
* any object we've already traversed, but may be
* reachable from an object we haven't gotten to yet.
* visit_reachable will eventually move gc back into
* young if that's so, and we'll see it again.*/next= gc->gc.gc_next;
gc_list_move(gc, unreachable);
gc->gc.gc_refs =GC_TENTATIVELY_UNREACHABLE;
}
gc=next;
}
}static intvisit_reachable(PyObject*op, PyGC_Head *reachable)
{if(PyObject_IS_GC(op)) {
PyGC_Head*gc =AS_GC(op);const Py_ssize_t gc_refs = gc->gc.gc_refs;if (gc_refs == 0) {/*This is in move_unreachable's 'young' list, but
* the traversal hasn't yet gotten to it. All
* we need to do is tell move_unreachable that it's
* reachable.*/gc->gc.gc_refs = 1;
}else if (gc_refs ==GC_TENTATIVELY_UNREACHABLE) {/*This had gc_refs = 0 when move_unreachable got
* to it, but turns out it's reachable after all.
* Move it back to move_unreachable's 'young' list,
* and move_unreachable will eventually get to it
* again.*/gc_list_move(gc, reachable);
gc->gc.gc_refs = 1;
}/*Else there's nothing to do.
* If gc_refs > 0, it must be in move_unreachable's 'young'
* list, and move_unreachable will eventually get to it.
* If gc_refs == GC_REACHABLE, it's either in some other
* generation so we don't care about it, or move_unreachable
* already dealt with it.
* If gc_refs == GC_UNTRACKED, it must be ignored.*/
else{
assert(gc_refs> 0
|| gc_refs ==GC_REACHABLE|| gc_refs ==GC_UNTRACKED);
}
}return 0;
}
View Code
分割完就得到垃圾回收目标对象，unreachable链表中的对象。
但是，并不是所有在unreachable链表中的对象都能安全回收。
当一个container对象，从类对象实例化出来的实例对象，定义了__del__方法时(python中称为finalizer)。当一个拥有finalizer的实例对象被销毁时，首先调用finalizer，因为__del__是python在对象销毁时进行资源释放的Hook机制。问题是，unreachable链表中都是循环引用对象，需要被销毁，其中有对象的finalizer引用了另一对象，python又不能保证销毁顺序。python将unreachable链表中拥有finalizer的PyInstanceObject都移到garbage的PyListObject对象中。
回收unreachable链表中的垃圾对象：
[gcmodule.c]static intgc_list_is_empty(PyGC_Head*list)
{return (list->gc.gc_next ==list);
}/*Break reference cycles by clearing the containers involved. This is
* tricky business as the lists can be changing and we don't know which
* objects may be freed. It is possible I screwed something up here.*/
static voiddelete_garbage(PyGC_Head*collectable, PyGC_Head *old)
{
inquiry clear;while (!gc_list_is_empty(collectable)) {
PyGC_Head*gc = collectable->gc.gc_next;
PyObject*op =FROM_GC(gc);
assert(IS_TENTATIVELY_UNREACHABLE(op));if (debug &DEBUG_SAVEALL) {
PyList_Append(garbage, op);
}else{if ((clear = Py_TYPE(op)->tp_clear) !=NULL) {
Py_INCREF(op);
clear(op);
Py_DECREF(op);
}
}if (collectable->gc.gc_next ==gc) {/*object is still alive, move it, it may die later*/gc_list_move(gc, old);
gc->gc.gc_refs =GC_REACHABLE;
}
}
}
对ob_refcnt下手，将unreachable链表中所有对象ob_refcnt变为0，引发对象销毁。
其中调用container对象的tp_clear操作，调整container对象中每个引用所用的对象的引用计数值，从而打破循环。
实际完成垃圾收集的collect：
[gcmodule.c]/*This is the main function. Read this to understand how the
* collection process works.*/
staticPy_ssize_t
collect(intgeneration)
{inti;
Py_ssize_t m= 0; /*# objects collected*/Py_ssize_t n= 0; /*# unreachable objects that couldn't be collected*/PyGC_Head*young; /*the generation we are examining*/PyGC_Head*old; /*next older generation*/PyGC_Head unreachable;/*non-problematic unreachable trash*/PyGC_Head finalizers;/*objects with, & reachable from, __del__*/PyGC_Head*gc;double t1 = 0.0;if (delstr ==NULL) {
delstr= PyString_InternFromString("__del__");if (delstr ==NULL)
Py_FatalError("gc couldn't allocate \"__del__\"");
}if (debug &DEBUG_STATS) {
PySys_WriteStderr("gc: collecting generation %d...\n",
generation);
PySys_WriteStderr("gc: objects in each generation:");for (i = 0; i < NUM_GENERATIONS; i++)
PySys_WriteStderr("%" PY_FORMAT_SIZE_T "d",
gc_list_size(GEN_HEAD(i)));
t1=get_time();
PySys_WriteStderr("\n");
}/*update collection and allocation counters*/
if (generation+1
generations[generation+1].count += 1;for (i = 0; i <= generation; i++)
generations[i].count= 0;/*merge younger generations with one we are currently collecting*/
for (i = 0; i < generation; i++) {
gc_list_merge(GEN_HEAD(i), GEN_HEAD(generation));
}/*handy references*/young=GEN_HEAD(generation);if (generation < NUM_GENERATIONS-1)
old= GEN_HEAD(generation+1);elseold=young;/*Using ob_refcnt and gc_refs, calculate which objects in the
* container set are reachable from outside the set (i.e., have a
* refcount greater than 0 when all the references within the
* set are taken into account).*/update_refs(young);
subtract_refs(young);/*Leave everything reachable from outside young in young, and move
* everything else (in young) to unreachable.
* NOTE: This used to move the reachable objects into a reachable
* set instead. But most things usually turn out to be reachable,
* so it's more efficient to move the unreachable things.*/gc_list_init(&unreachable);
move_unreachable(young,&unreachable);/*Move reachable objects to next generation.*/
if (young !=old) {if (generation == NUM_GENERATIONS - 2) {
long_lived_pending+=gc_list_size(young);
}
gc_list_merge(young, old);
}else{/*We only untrack dicts in full collections, to avoid quadratic
dict build-up. See issue #14775.*/untrack_dicts(young);
long_lived_pending= 0;
long_lived_total=gc_list_size(young);
}/*All objects in unreachable are trash, but objects reachable from
* finalizers can't safely be deleted. Python programmers should take
* care not to create such things. For Python, finalizers means
* instance objects with __del__ methods. Weakrefs with callbacks
* can also call arbitrary Python code but they will be dealt with by
* handle_weakrefs().*/gc_list_init(&finalizers);
move_finalizers(&unreachable, &finalizers);/*finalizers contains the unreachable objects with a finalizer;
* unreachable objects reachable *from* those are also uncollectable,
* and we move those into the finalizers list too.*/move_finalizer_reachable(&finalizers);/*Collect statistics on collectable objects found and print
* debugging information.*/
for (gc = unreachable.gc.gc_next; gc != &unreachable;
gc= gc->gc.gc_next) {
m++;if (debug &DEBUG_COLLECTABLE) {
debug_cycle("collectable", FROM_GC(gc));
}
}/*Clear weakrefs and invoke callbacks as necessary.*/m+= handle_weakrefs(&unreachable, old);/*Call tp_clear on objects in the unreachable set. This will cause
* the reference cycles to be broken. It may also cause some objects
* in finalizers to be freed.*/delete_garbage(&unreachable, old);/*Collect statistics on uncollectable objects found and print
* debugging information.*/
for (gc =finalizers.gc.gc_next;
gc!= &finalizers;
gc= gc->gc.gc_next) {
n++;if (debug &DEBUG_UNCOLLECTABLE)
debug_cycle("uncollectable", FROM_GC(gc));
}if (debug &DEBUG_STATS) {double t2 =get_time();if (m == 0 && n == 0)
PySys_WriteStderr("gc: done");elsePySys_WriteStderr("gc: done,"
"%" PY_FORMAT_SIZE_T "d unreachable,"
"%" PY_FORMAT_SIZE_T "d uncollectable",
n+m, n);if (t1 &&t2) {
PySys_WriteStderr(", %.4fs elapsed", t2-t1);
}
PySys_WriteStderr(".\n");
}/*Append instances in the uncollectable set to a Python
* reachable list of garbage. The programmer has to deal with
* this if they insist on creating this type of structure.*/(void)handle_finalizers(&finalizers, old);/*Clear free list only during the collection of the highest
* generation*/
if (generation == NUM_GENERATIONS-1) {
clear_freelists();
}if(PyErr_Occurred()) {if (gc_str ==NULL)
gc_str= PyString_FromString("garbage collection");
PyErr_WriteUnraisable(gc_str);
Py_FatalError("unexpected exception during garbage collection");
}return n+m;
}
View Code
python中的垃圾收集机制完全是为了处理循环引用而设计的，几乎大多数对象创建时都会被纳入垃圾收集机制的监控中。并且，正常的引用计数就能销毁一个被纳入垃圾收集机制监控的对象。
python很多对象挂在垃圾收集监控的链表上，但大多情况是引用计数在维护这些对象。对引用计数无能为力的循环引用，垃圾收集机制才起作用。而垃圾收集机制只处理引用计数不为0的情况：一是被程序使用的对象(不能被回收)，二是循环引用对象。因此垃圾回收机制只能处理循环引用中的对象。
还有一点，PyObject_GC_New底层是以之前剖析的PyObject_Malloc作为真正申请内存的接口的，大多数情况下Python都在使用内存池。而本书中剖析过得最大的对象PyTypeObject也不超过200个字节，小于256个字节，故也使用内存池。因此可将垃圾收集和内存管理融为一体。
4.5、python 中 的gc模块
python中通过gc模块提供了观察和手动实用gc机制的接口。
具体打开python，动手实验。
2707
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
