话说Python内存管理机制篇三之内存分配策略(二)

by LauCyun Aug 01,2017 17:50:34 17,412 views

上篇谈Python内存管理机制篇二之内存分配策略(一)讲了blockpool,接着继续讲arena

3 arena

在Python中,多个pool的集合就是一个arena

3.1 arena大小

上篇谈Python内存管理机制篇二之内存分配策略(一)讲了,pool的大小的默认值为4kb,同样每个arena的大小也有一个默认的值。

在Python3.5中,每个arena的大小的默认值为256kb,那么一个arena有多少个pool呢?很容易算出来,一个arenaARENA_SIZE / POOL_SIZEpool

[Objects/obmalloc.c]

#define ARENA_SIZE              (256 << 10)     /* 256KB */

3.2 arena结构

好了,说了那么多,那么arena是个神马鬼呢?先来看如下代码:

[Objects/obmalloc.c]

/* Record keeping for arenas. */
struct arena_object {
    /* The address of the arena, as returned by malloc.  Note that 0
     * will never be returned by a successful malloc, and is used
     * here to mark an arena_object that doesn't correspond to an
     * allocated arena.
     */
    uptr address;

    /* Pool-aligned pointer to the next pool to be carved off. */
    block* pool_address;

    /* The number of available pools in the arena:  free pools + never-
     * allocated pools.
     */
    uint nfreepools;

    /* The total number of pools in the arena, whether or not available. */
    uint ntotalpools;

    /* Singly-linked list of available pools. */
    struct pool_header* freepools;

    /* Whenever this arena_object is not associated with an allocated
     * arena, the nextarena member is used to link all unassociated
     * arena_objects in the singly-linked `unused_arena_objects` list.
     * The prevarena member is unused in this case.
     *
     * When this arena_object is associated with an allocated arena
     * with at least one available pool, both members are used in the
     * doubly-linked `usable_arenas` list, which is maintained in
     * increasing order of `nfreepools` values.
     *
     * Else this arena_object is associated with an allocated arena
     * all of whose pools are in use.  `nextarena` and `prevarena`
     * are both meaningless in this case.
     */
    struct arena_object* nextarena;
    struct arena_object* prevarena;
};

从上面的源码中不难获知,一个arena是有arena_objectpool集合组成。如图1所示:


图1 arena结构

arena_object仅仅是一个arena的一部分,就像pool_header只是的pool一部分一样,看上去作用是一样的,但是arena_object管理的内存和pool_header管理的内存是有不一样。神马,有不一样?具体的请先看图2


图2 pool和arena的内存布局区别

差别就是:pool_header管理的内存与pool_header自身是一块连续的内存arena_object管理的内存与自身是分离的

看上去没什么大不了的,但是这后面隐藏着这样一个事实:

pool_header被申请时,它所管理的block集合的内存一定也被申请了;但是arena_object被申请时,它所管理的pool集合的内存则没有被申请。换句话说,arena_objectpool集合在某一时刻需要建立联系。注意,这个建立联系的时刻是一个关键的时刻,Python从这个时刻一刀切下,将一个arena_object切分为两种状态:未使用(没有建立联系)和可用(建立了联系)。

3.3 arena两种状态:未使用(没有建立联系)和可用(建立了联系)

当一个arenaarena_object没有与pool集合建立联系时,这时的arena处于“未使用”状态;一旦建立了联系,这时arena就转换到了“可用”状态。对于每一种状态,都有一个arena的链表。

[Objects/obmalloc.c]

/* The head of the singly-linked, NULL-terminated list of available
 * arena_objects.
 */
static struct arena_object* unused_arena_objects = NULL;

/* The head of the doubly-linked, NULL-terminated at each end, list of
 * arena_objects associated with arenas that have pools available.
 */
static struct arena_object* usable_arenas = NULL;

有两个链表:

  • “未使用”的arena的链表表头是unused_arena_objectsarenaarena之间通过nextarena连接,是一个单向链表。
  • “可用”的arena的链表表头是usable_arenasarenaarena之间通过nextarenaprevarena连接,是一个双向链表。

不能理解?好吧,画一张图来展示一下,如图3是某一时刻arena的一个可能状态:


图3 arena在某一时刻的可能状态

3.4  申请新的arena

在了解arena的创建之前,先看一下arena创建时相关的一些参数定义:

[Objects/obmalloc.c]

/* Array of objects used to track chunks of memory (arenas). */
// arena_object 数组
static struct arena_object* arenas = NULL;
/* Number of slots currently allocated in the `arenas` vector. */
// 当前arenas中管理的arena_object的个数, 初始化时为0
static uint maxarenas = 0;

/* The head of the singly-linked, NULL-terminated list of available
 * arena_objects.
 */
// “未使用”的arena_object链表
static struct arena_object* unused_arena_objects = NULL;

/* The head of the doubly-linked, NULL-terminated at each end, list of
 * arena_objects associated with arenas that have pools available.
 */
// “可用”的arena_object链表
static struct arena_object* usable_arenas = NULL;

/* How many arena_objects do we initially allocate?
 * 16 = can allocate 16 arenas = 16 * ARENA_SIZE = 4MB before growing the
 * `arenas` vector.
 */
// 初始化时需要申请的arena_object的个数
#define INITIAL_ARENA_OBJECTS 16

/* Number of arenas allocated that haven't been free()'d. */
// 已分配的还没有被释放的arena_object的个数
static size_t narenas_currently_allocated = 0;

/* Total number of times malloc() called to allocate an arena. */
// 调用malloc()的次数
static size_t ntimes_arena_allocated = 0;
/* High water mark (max value ever seen) for narenas_currently_allocated. */
// narenas_currently_allocated的最大值
static size_t narenas_highwater = 0;

Python运行期间,它会通过new_arena来创建一个arena,先看一下new_arena的过程:

[Objects/obmalloc.c]

/* Allocate a new arena.  If we run out of memory, return NULL.  Else
 * allocate a new arena, and return the address of an arena_object
 * describing the new arena.  It's expected that the caller will set
 * `usable_arenas` to the return value.
 */
static struct arena_object*
new_arena(void)
{
    struct arena_object* arenaobj;
    uint excess;        /* number of bytes above pool alignment */
    void *address;

#ifdef PYMALLOC_DEBUG
    if (Py_GETENV("PYTHONMALLOCSTATS"))
        _PyObject_DebugMallocStats(stderr);
#endif
    //代码[1]:判断unused_arena_objects为NULL,为NULL是则没有“未使用的”arena_object列表,则需要扩充
    if (unused_arena_objects == NULL) {
        uint i;
        uint numarenas;
        size_t nbytes;

        /* Double the number of arena objects on each allocation.
         * Note that it's possible for `numarenas` to overflow.
         */
        //代码[2]:确定本次需要申请的arena_object的个数(第一次为16,之后每次翻倍)
        numarenas = maxarenas ? maxarenas << 1 : INITIAL_ARENA_OBJECTS;
        if (numarenas <= maxarenas)
            return NULL;                /* overflow(溢出了) */
#if SIZEOF_SIZE_T <= SIZEOF_INT
        if (numarenas > PY_SIZE_MAX / sizeof(*arenas))
            return NULL;                /* overflow(溢出了) */
#endif
        nbytes = numarenas * sizeof(*arenas);
        //代码[3]:使用 PyMem_RawRealloc 申请内存
        arenaobj = (struct arena_object *)PyMem_RawRealloc(arenas, nbytes);
        if (arenaobj == NULL)
            return NULL;
        arenas = arenaobj;

        /* We might need to fix pointers that were copied.  However,
         * new_arena only gets called when all the pages in the
         * previous arenas are full.  Thus, there are *no* pointers
         * into the old array. Thus, we don't have to worry about
         * invalid pointers.  Just to be sure, some asserts:
         */
        assert(usable_arenas == NULL);
        assert(unused_arena_objects == NULL);

        //代码[4]:初始化新申请的arena_object,并将其放入unused_arena_objects链表中
        /* Put the new arenas on the unused_arena_objects list. */
        for (i = maxarenas; i < numarenas; ++i) {
            arenas[i].address = 0;              /* mark as unassociated */
            arenas[i].nextarena = i < numarenas - 1 ?
                                   &arenas[i+1] : NULL;
        }

        /* Update globals. */
        unused_arena_objects = &arenas[maxarenas];
        maxarenas = numarenas;
    }

    /* Take the next available arena object off the head of the list. */
    //代码[5]:从unused_arena_objects链表中取出一个“未使用”的arena_object
    assert(unused_arena_objects != NULL);
    arenaobj = unused_arena_objects;
    unused_arena_objects = arenaobj->nextarena; // 更新unused_arena_objects指针,指向下一个arena
    
    //代码[6]:申请arena_object管理的内存
    assert(arenaobj->address == 0);
    address = _PyObject_Arena.alloc(_PyObject_Arena.ctx, ARENA_SIZE);
    if (address == NULL) {
        /* The allocation failed: return NULL after putting the
         * arenaobj back.
         */
        arenaobj->nextarena = unused_arena_objects;
        unused_arena_objects = arenaobj;
        return NULL;
    }
    arenaobj->address = (uptr)address;
    
    //代码[7]:设置初始化相关参数
    ++narenas_currently_allocated;
    ++ntimes_arena_allocated;
    if (narenas_currently_allocated > narenas_highwater)
        narenas_highwater = narenas_currently_allocated;
    
    //代码[8]:设置pool集合的相关信息
    arenaobj->freepools = NULL;
    /* pool_address <- first pool-aligned address in the arena
       nfreepools <- number of whole pools that fit after alignment */
    arenaobj->pool_address = (block*)arenaobj->address;
    arenaobj->nfreepools = ARENA_SIZE / POOL_SIZE;
    assert(POOL_SIZE * arenaobj->nfreepools == ARENA_SIZE);
    // 将pool的起始地址调整为系统页的边界
    // 申请到256KB, 放弃了一些内存, 而将可使用的内存边界pool_address调整到了与系统页对齐
    excess = (uint)(arenaobj->address & POOL_SIZE_MASK);
    if (excess != 0) {
        --arenaobj->nfreepools;
        arenaobj->pool_address += POOL_SIZE - excess;
    }
    arenaobj->ntotalpools = arenaobj->nfreepools;

    return arenaobj;
}

为了更好理解new_arena的过程,画了图4


图4 arena在某一时刻的可能状态

在代码[1]处,Python首先会检查当前unused_arena_objects链表中是否还有“未使用”的arena,检查结果将会决定下一步的动作。

3.4.1 从unused_arena_objects中取一个arena初始化

假如有一个unused_arena_objects,如图5所示:


图5 从unused_arena_objects中取出一个arena

如果unused_arena_objects中还有“未使用”的arena,那么Python将直接在代码[5]处开始,从unused_arena_objects中取出一个arena,接着unused_arena_objects指向下一个arena,如图6所示。


图6 从unused_arena_objects中取出一个arena

然后在代码[6]处,Python申请一块为ARENA_SIZE(256KB)的内存,将内存的地址赋给arenaaddress,此时arena_object就和pool集合建立了联系,也就是成为“可用”的arena了,如图7所示。


图7 申请arena_object管理的内存

所以,到这里这个arena就和unused_arena_objects脱离了关系,被usable_arenas这个组织接收了。

在代码[7]中,Python设置了narenas_currently_allocatedntimes_arena_allocatednarenas_highwater参数。

在代码[8]中,Python设置了一些arena用于维护pool集合的信息。

  • 对申请到的ARENA_SIZE(256KB)内存进行处理,放弃了一些内存,将可用的内存边界pool_addreess调整到了与系统页对齐。
  • freepools设置为NULL,前面讲pool的时候知道,freepools是要等到有pool释放后才起作用。

实际上,pool集合所占用的ARENA_SIZE(256KB)内存进行边界对齐后,是交给pool_addreess来维护。如8所示。


8 设置arena用于维护pool集合的信息

好了,到这为此一个arena的初始化基本上完成,但是还有一个待解决的问题:usable_arenas组织什么时候才能接收这个arena呢?这是后话cheeky

3.4.2 扩充unused_arena_objects

上面讲了如何从unused_arena_objects中取出一个arena进行初始化。

现在回到代码[1]处的判断,如果unused_arena_objectsNULL,则需要扩大arena的集合(小块内存内存池)。Python在内部有一个maxarenas的变量来维护arenas中存储的arena_object个数。在代码[2]中处,Python将待申请的arena_object个数设置为当前arena_object个数(maxarenas)的两倍。当然,在首次初始化时,maxarenas为0,这时将numarenas初始化为16。之后,Python会对numarenas进行检查是否溢出。

如果检查通过的话,则在代码[3]处,通过PyMem_RawRealloc来扩大内存,并在代码[4]处对新申请的arena_object进行设置。特别要注意的是那个貌似毫不起眼的address,代码[4]处将新申请的arenaaddress一律设置为0,这也是一个arena是处于“未使用”状态还是“可用”状态的重要标识。在代码[6]处,一旦arenaarena_objectpool集合建立了联系,那么address就变成了非0。

最后代码[4]处,将新申请的arenas赋值给了unused_arena_objects,这样一来,就又有了“未使用”的arena了。接下来就是回到3.4.1中了。

Tags