Skip to content

gh-142516: Forward-port generational GC #148687

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
zanieb wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

gh-142516: Forward-port generational GC #148687

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 20 additions & 39 deletions Doc/library/gc.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,18 +37,11 @@ The :mod:`!gc` module provides the following functions:

.. function:: collect(generation=2)

Perform a collection. The optional argument *generation*
With no arguments, run a full collection. The optional argument *generation*
may be an integer specifying which generation to collect (from 0 to 2). A
:exc:`ValueError` is raised if the generation number is invalid. The sum of
collected objects and uncollectable objects is returned.

Calling ``gc.collect(0)`` will perform a GC collection on the young generation.

Calling ``gc.collect(1)`` will perform a GC collection on the young generation
and an increment of the old generation.

Calling ``gc.collect(2)`` or ``gc.collect()`` performs a full collection

The free lists maintained for a number of built-in types are cleared
whenever a full collection or collection of the highest generation (2)
is run. Not all items in some free lists may be freed due to the
Expand All @@ -57,9 +50,6 @@ The :mod:`!gc` module provides the following functions:
The effect of calling ``gc.collect()`` while the interpreter is already
performing a collection is undefined.

.. versionchanged:: 3.14
``generation=1`` performs an increment of collection.


.. function:: set_debug(flags)

Expand All @@ -77,18 +67,12 @@ The :mod:`!gc` module provides the following functions:


Returns a list of all objects tracked by the collector, excluding the list
returned. If *generation* is not ``None``, return only the objects as follows:

* 0: All objects in the young generation
* 1: No objects, as there is no generation 1 (as of Python 3.14)
* 2: All objects in the old generation
returned. If *generation* is not ``None``, return only the objects tracked by
the collector that are in that generation.

.. versionchanged:: 3.8
New *generation* parameter.

.. versionchanged:: 3.14
Generation 1 is removed

.. audit-event:: gc.get_objects generation gc.get_objects

.. function:: get_stats()
Expand Down Expand Up @@ -124,33 +108,30 @@ The :mod:`!gc` module provides the following functions:
Set the garbage collection thresholds (the collection frequency). Setting
*threshold0* to zero disables collection.

The GC classifies objects into two generations depending on whether they have
survived a collection. New objects are placed in the young generation. If an
object survives a collection it is moved into the old generation.

In order to decide when to run, the collector keeps track of the number of object
The GC classifies objects into three generations depending on how many
collection sweeps they have survived. New objects are placed in the youngest
generation (generation ``0``). If an object survives a collection it is moved
into the next older generation. Since generation ``2`` is the oldest
generation, objects in that generation remain there after a collection. In
order to decide when to run, the collector keeps track of the number object
allocations and deallocations since the last collection. When the number of
allocations minus the number of deallocations exceeds *threshold0*, collection
starts. For each collection, all the objects in the young generation and some
fraction of the old generation is collected.
starts. Initially only generation ``0`` is examined. If generation ``0`` has
been examined more than *threshold1* times since generation ``1`` has been
examined, then generation ``1`` is examined as well.
With the third generation, things are a bit more complicated,
see `Garbage collector design <https://devguide.python.org/garbage_collector>`_
for more information.

.. note::
In the free-threaded build, the cycle collector is not generational.
Collections operate over the entire tracked heap.

In the free-threaded build, the increase in process memory usage is also
checked before running the collector. If the memory usage has not increased
checked before running the collector. If the memory usage has not increased
by 10% since the last collection and the net number of object allocations
has not exceeded 40 times *threshold0*, the collection is not run.

The fraction of the old generation that is collected is **inversely** proportional
to *threshold1*. The larger *threshold1* is, the slower objects in the old generation
are collected.
For the default value of 10, 1% of the old generation is scanned during each collection.

*threshold2* is ignored.

See `Garbage collector design <https://devguide.python.org/garbage_collector>`_ for more information.

.. versionchanged:: 3.14
*threshold2* is ignored


.. function:: get_count()

Expand Down
21 changes: 9 additions & 12 deletions Include/internal/pycore_gc.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,6 @@ extern void _Py_ScheduleGC(PyThreadState *tstate);
extern void _Py_TriggerGC(struct _gc_runtime_state *gcstate);
#endif


/* Tell the GC to track this object.
*
* The object must not be tracked by the GC.
Expand All @@ -220,7 +219,7 @@ extern void _Py_TriggerGC(struct _gc_runtime_state *gcstate);
* ob_traverse method.
*
* Internal note: interp->gc.generation0->_gc_prev doesn't have any bit flags
* because it's not object header. So we don't use _PyGCHead_PREV() and
* because it's not an object header. So we don't use _PyGCHead_PREV() and
* _PyGCHead_SET_PREV() for it to avoid unnecessary bitwise operations.
*
* See also the public PyObject_GC_Track() function.
Expand All @@ -245,16 +244,15 @@ static inline void _PyObject_GC_TRACK(
filename, lineno, __func__);

struct _gc_runtime_state *gcstate = &_PyInterpreterState_GET()->gc;
PyGC_Head *generation0 = &gcstate->young.head;
PyGC_Head *last = (PyGC_Head*)(generation0->_gc_prev);
PyGC_Head *generation0 = gcstate->generation0;
PyGC_Head *last = (PyGC_Head *)(generation0->_gc_prev);
_PyGCHead_SET_NEXT(last, gc);
_PyGCHead_SET_PREV(gc, last);
uintptr_t not_visited = 1 ^ gcstate->visited_space;
gc->_gc_next = ((uintptr_t)generation0) | not_visited;
_PyGCHead_SET_NEXT(gc, generation0);
generation0->_gc_prev = (uintptr_t)gc;
gcstate->young.count++; /* number of tracked GC objects */
gcstate->heap_size++;
if (gcstate->young.count > gcstate->young.threshold) {
/* gh-139951: count tracked GC objects, not all GC-capable allocations. */
gcstate->generations[0].count++; /* number of tracked GC objects */
if (gcstate->generations[0].count > gcstate->generations[0].threshold) {
_Py_TriggerGC(gcstate);
}
#endif
Expand Down Expand Up @@ -292,10 +290,9 @@ static inline void _PyObject_GC_UNTRACK(
gc->_gc_next = 0;
gc->_gc_prev &= _PyGC_PREV_MASK_FINALIZED;
struct _gc_runtime_state *gcstate = &_PyInterpreterState_GET()->gc;
if (gcstate->young.count > 0) {
gcstate->young.count--;
if (gcstate->generations[0].count > 0) {
gcstate->generations[0].count--;
}
gcstate->heap_size--;
#endif
}

Expand Down
11 changes: 9 additions & 2 deletions Include/internal/pycore_interp_structs.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,9 +243,16 @@ struct _gc_runtime_state {
/* Is automatic collection enabled? */
int enabled;
int debug;
/* linked lists of container objects */

/* Generational GC state used in GIL builds. */
struct gc_generation generations[NUM_GENERATIONS];
PyGC_Head *generation0;
struct gc_generation_stats generation_stats_gen[NUM_GENERATIONS];

/* Incremental/free-threaded GC state. */
struct gc_generation young;
struct gc_generation old[2];

/* a permanent generation which won't be collected */
struct gc_generation permanent_generation;
struct gc_stats *generation_stats;
Expand All @@ -265,7 +272,6 @@ struct _gc_runtime_state {
int visited_space;
int phase;

#ifdef Py_GIL_DISABLED
/* This is the number of objects that survived the last full
collection. It approximates the number of long lived objects
tracked by the GC.
Expand All @@ -278,6 +284,7 @@ struct _gc_runtime_state {
the first time. */
Py_ssize_t long_lived_pending;

#ifdef Py_GIL_DISABLED
/* True if gc.freeze() has been used. */
int freeze_active;

Expand Down
5 changes: 5 additions & 0 deletions Include/internal/pycore_runtime_init.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,11 @@ extern PyTypeObject _PyExc_MemoryError;
}, \
.gc = { \
.enabled = 1, \
.generations = { \
{ .threshold = 2000, }, \
{ .threshold = 10, }, \
{ .threshold = 10, }, \
}, \
.young = { .threshold = 2000, }, \
.old = { \
{ .threshold = 10, }, \
Expand Down
Loading
Loading