gh-159111: Update vendored mimalloc to v2.5.2 by corona10 · Pull Request #159112 · python/cpython

corona10 · 2026-10-10T11:24:38Z

Issue: Upgrade mimalloc version from 2.1.2 to 2.5.2 #159111

bedevere-bot · 2026-10-10T11:25:18Z

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 2649e2b 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

Upgrade mimalloc from v2.1.2 to v2.5.2 as-is from upstream. v2.1.2: https://github.com/microsoft/mimalloc/releases/tag/v2.1.2 v2.5.2: https://github.com/microsoft/mimalloc/releases/tag/v2.5.2

bedevere-bot · 2026-10-10T11:30:47Z

🤖 New build scheduled with the buildbot fleet by @corona10 for commit f81e6c2 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

corona10 · 2026-10-10T11:34:29Z

@mpage, could you trigger a free-threading benchmark for this change? :)

read-the-docs-community · 2026-10-10T11:34:59Z

Documentation build overview

📚 cpython-previews | 🛠️ Build #35117179 | 📁 Comparing 8b9cb49 against main (88fd5fe)

🔍 Preview build

16 files changed · ± 16 modified

± Modified

bedevere-bot · 2026-10-10T11:50:50Z

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 222601b 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

corona10 · 2026-10-10T11:59:00Z

I will fix MSAN later:

==11983==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55e368c48377 in _mi_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:158:3
    #1 0x55e368c48377 in _mi_heap_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/heap.c:281:10
    #2 0x55e368c48377 in mi_heap_main_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:220:29
    #3 0x55e368c49eaf in _mi_auto_process_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:568:3
    #4 0x7f8aa762aa53 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2aa53) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #5 0x55e36886ee24 in _start (/home/runner/work/cpython/cpython/Programs/_freeze_module+0xb4e24) (BuildId: 5e5dc085937d530e3b49b31b09d53deb8664a51d)

  Uninitialized value was stored to memory at
    #0 0x55e368c8b8a0 in chacha_block /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:67:20
    #1 0x55e368c47beb in chacha_next32 /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:83:5
    #2 0x55e368c47beb in _mi_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:154:22
    #3 0x55e368c47beb in _mi_heap_random_next /home/runner/work/cpython/cpython/Objects/mimalloc/heap.c:281:10
    #4 0x55e368c47beb in mi_heap_main_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:220:29
    #5 0x55e368c49eaf in _mi_auto_process_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:568:3
    #6 0x7f8aa762aa53 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2aa53) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #7 0x55e36886ee24 in _start (/home/runner/work/cpython/cpython/Programs/_freeze_module+0xb4e24) (BuildId: 5e5dc085937d530e3b49b31b09d53deb8664a51d)

  Uninitialized value was stored to memory at
    #0 0x55e368c5d150 in chacha_init /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:109:23
    #1 0x55e368c5d150 in mi_random_init_ex /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:201:3
    #2 0x55e368c47b57 in _mi_random_init /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:205:3
    #3 0x55e368c47b57 in mi_heap_main_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:218:7
    #4 0x55e368c49eaf in _mi_auto_process_init /home/runner/work/cpython/cpython/Objects/mimalloc/init.c:568:3
    #5 0x7f8aa762aa53 in __libc_start_main (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2aa53) (BuildId: 4ec69afdf3da96ce7388640dcd27c6d0c1b256ef)
    #6 0x55e36886ee24 in _start (/home/runner/work/cpython/cpython/Programs/_freeze_module+0xb4e24) (BuildId: 5e5dc085937d530e3b49b31b09d53deb8664a51d)

  Uninitialized value was created by an allocation of 'key' in the stack frame
    #0 0x55e368c5cbad in mi_random_init_ex /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:181:3

SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/runner/work/cpython/cpython/Objects/mimalloc/random.c:158:3 in _mi_random_next
Exiting

Most of the previous downstream changes were upstreamed in mimalloc v2.1.3 - v2.1.7 (heap tags, _mi_heap_init/_mi_tld_init, MI_PRIM_THREAD_ID, sub-processes, heap walk optimizations, platform fixes) and are dropped. Remaining CPython-specific changes, all marked with a "CPython" comment: - QSBR delayed page freeing for free-threaded builds (pythongh-115103, pythongh-145615): mi_page_t/mi_heap_t fields and _PyMem_mi_* hooks in page.c, heap.c and segment.c. - debug_offset: preserve the first bytes of a PyObject (ob_tid, refcount fields) when filling freed or uninitialized memory in debug builds (pythongh-115491). - Atomic mi_block_t.next loads/stores to avoid TSAN reports (pythongh-129748). - Relax the owning thread assertion in _mi_free_delayed_block for free-threaded builds, since the GC traverses heaps of other threads. - Call _mi_heap_delayed_free_partial() in mi_heap_visit_blocks() (pythongh-112532). - Only count reclaimed pages up to MI_MEDIUM_OBJ_SIZE_MAX as a "right page" in mi_segment_reclaim (pythongh-115103). - Rename the weak _ZSt15get_new_handlerv symbol and disable the mi_is_in_heap_region debug warning. - Zero-initialize _mi_page_empty instead of a positional initializer.

mpage · 2026-10-11T01:27:23Z

Here's the benchmarking run. Will report results when it completes.

corona10 · 2026-10-11T21:24:42Z

Benchmark result: https://github.com/facebookexperimental/free-threading-benchmarking/blob/190b19ecfdfcde5ce50d5793f718e42422dcf5f7/results/bm-20261010-3.16.0a0-91f4f86-NOGIL/README.md#vs-base

Hmm.. higher memory usage?

corona10 · 2026-10-11T21:31:01Z

Let me check where the memory pressure came from.

corona10 · 2026-10-11T22:52:00Z

Okay this is because, MIMALLOC_ALLOW_THP=1 become by default.
https://github.com/microsoft/mimalloc/blob/v2.5.2/readme.md#environment-options

corona10 · 2026-10-11T23:05:19Z

@mpage Would you like to re-trigger the benchmark? The latest commit will resolve the higher memory usage issue.

corona10 · 2026-10-11T23:17:48Z

cc @daanx

methane · 2026-10-12T15:31:28Z

Based on Codex review, following compatibility fixes are still missing:

Honor QSBR during forced page abandonment. _mi_page_force_abandon() calls _mi_page_free() directly for empty pages, even if they are still waiting for a QSBR grace period. It should use _PyMem_mi_page_maybe_free() and abandon the page if it cannot yet be freed. I reproduced an assertion failure with a C reproducer using a page awaiting QSBR. (Fix)
Synchronize abandoned-segment iteration across interpreters. Stopping one interpreter does not stop other interpreters from scanning the shared arena bitmap. A scanner temporarily clears a segment’s bit before checking its subproc, so another interpreter’s GC can miss that segment. The fork enables MI_VISIT_ABANDONED and uses the upstream visitor’s locking. Bypassing the option check requires equivalent synchronization. (Fix)
Disable guarded allocations until CPython supports their layout. The fork defines MI_GUARDED=0. With this PR’s free-threaded debug build, MIMALLOC_GUARDED_SAMPLE_RATE=1 ./python -S -c 'pass' crashes with SIGSEGV. (Fix)
Collect pending frees before reporting abandoned-page usage. In mi_segment_visit_page(), _mi_page_free_collect() should run before _mi_heap_area_init(), including when visit_blocks is false. Otherwise, sys.getallocatedblocks() counts already-freed blocks. The fork’s regression test fails on this PR: freeing 20,000 objects allocated by an exited thread produces no decrease in the count. (Fix)
Release subproc metadata when it is safe. Each interpreter allocates a subproc, but there is no corresponding cleanup. The fork calls mi_subproc_delete() during interpreter teardown only when abandoned_count == 0. Unconditional deletion would be unsafe because surviving segments may still reference the metadata. (Fix)

mpage · 2026-10-12T15:54:10Z

@corona10 - New run

corona10 · 2026-10-12T16:00:25Z

@methane

Release subproc metadata when it is safe.

It's already happen in the current mimalloc, I think this could be of scope this PR. But let me check again.

We may cover other stuff in separated PR if they are not the scope this PR too.

corona10 · 2026-10-13T01:10:52Z

@corona10 - New run

Looks like it is failed to upload benchmark data :( because of internal error.

methane · 2026-10-13T03:03:21Z

    _Py_brc_init_state(interp);
+
+    interp->mimalloc.subproc = mi_subproc_new();
+    if (interp->mimalloc.subproc == NULL) {


this part is introduced in this PR so free() part should be too.

FYI, this leak already exists on main (free-threaded build).

import gc, resource, sys from concurrent import interpreters def rss_mib(): rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss return rss / (1024 * 1024) if sys.platform == "darwin" else rss / 1024 N = 100 interpreters.create().close() # warm up gc.collect(); before = rss_mib() for _ in range(N): interp = interpreters.create() interp.exec("x = [object() for _ in range(20000)]") interp.close() gc.collect(); after = rss_mib() print(f"RSS {before:.1f} -> {after:.1f} MiB, {(after - before) / N * 1024:.0f} KiB per interpreter")

On main (c6c0608):

RSS 22.9 -> 637.4 MiB, 6292 KiB per interpreter

When the last thread of an interpreter exits, its abandoned segments are never
reclaimed because no thread of that interpreter is left. So RSS keeps growing.

In this PR, the abandoned pool is replaced by interp->mimalloc.subproc, and the
situation is the same as before.

So I thought that it is okay to leave it as is.
But let me adding freeing logic at this change.

FYI, abandoned_count == 0 never reach in my local experiment. So I think that it will not actually execute in practice.

corona10 · 2026-10-13T12:35:09Z

Collect pending frees before reporting abandoned-page usage. In mi_segment_visit_page(), _mi_page_free_collect() should run before _mi_heap_area_init(), including when visit_blocks is false. Otherwise, sys.getallocatedblocks() counts already-freed blocks. The fork’s regression test fails on this PR: freeing 20,000 objects allocated by an exited thread produces no decrease in the count. (Fix)

@daanx
It's more like an upstream bug.
Do you prefer to send a patch to upstream?

corona10 · 2026-10-13T12:37:35Z

@methane

I've addressed most of your suggestions except for Release subproc metadata when it is safe.
Let me find a better solution for this. (not sure that current suggested fix is good)

Thank you for the suggestion.

corona10 · 2026-10-13T12:45:51Z

+    mi_subproc_t *subproc = interp->mimalloc.subproc;
+    if (subproc != NULL) {
+        // Delete the sub-process only when no abandoned segment points to it.
+        if (mi_atomic_load_relaxed(&subproc->abandoned_count) == 0) {


@methane

FYI: #159112 (comment)

I'm still not sure it will execute in practice, but I think it would be fine since this issue has already happened in the main branch.
But having deletion logic as an option is good convention.

We can solve this issue in a separate PR.

bedevere-bot · 2026-10-13T12:46:51Z

🤖 New build scheduled with the buildbot fleet by @corona10 for commit 8b9cb49 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F159112%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

mpage · 2026-10-13T15:43:40Z

@corona10 - Kicked off another run. Fingers crossed this one works.

mpage · 2026-10-13T18:01:12Z

@corona10 - New results

corona10 · 2026-10-13T21:55:04Z

@mpage

Okay benchmark result now looks good to me since there is no significant change

fork: corona10
ref: gh_159111
machine: linux-x86_64
commit hash: 8b9cb49
commit date: 2026-10-13
overall geometric mean: 1.003x faster
HPT reliability: 97.78%
HPT 99th percentile: 1.00x faster
Memory change: 1.00x

Would you like to review this PR?

bedevere-app Bot added the type-feature A feature request or enhancement label Oct 10, 2026

bedevere-app Bot mentioned this pull request Oct 10, 2026

Upgrade mimalloc version from 2.1.2 to 2.5.2 #159111

Open

corona10 self-assigned this Oct 10, 2026

corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026

corona10 marked this pull request as ready for review October 10, 2026 11:25

corona10 requested review from a team, AA-Turner, FFY00, ZeroIntensity, emmatyping, ericsnowcurrently, erlend-aasland and itamaro as code owners October 10, 2026 11:25

bedevere-app Bot added the awaiting core review label Oct 10, 2026

corona10 requested a review from mpage October 10, 2026 11:25

Update vendored mimalloc to v2.5.2

4180f6d

Upgrade mimalloc from v2.1.2 to v2.5.2 as-is from upstream. v2.1.2: https://github.com/microsoft/mimalloc/releases/tag/v2.1.2 v2.5.2: https://github.com/microsoft/mimalloc/releases/tag/v2.5.2

corona10 force-pushed the gh-159111 branch from 2649e2b to f81e6c2 Compare October 10, 2026 11:29

corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026

corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026

corona10 force-pushed the gh-159111 branch from ceb117e to 5fb37d1 Compare October 10, 2026 13:42

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 10, 2026

corona10 added 2 commits October 12, 2026 08:03

Disable THP

488b485

Merge remote-tracking branch 'upstream/main' into pythongh-159111

fea8db6

methane reviewed Oct 13, 2026

View reviewed changes

corona10 added 5 commits October 13, 2026 21:01

Address code review

f561f9f

Address code review

1ed1490

Address code review

c711d85

nit

b051985

Address code review

471d7da

corona10 requested a review from methane October 13, 2026 12:36

Address code review

8b9cb49

corona10 commented Oct 13, 2026

View reviewed changes

corona10 added the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 13, 2026

bedevere-bot removed the 🔨 test-with-refleak-buildbots Test PR w/ refleak buildbots; report in status section label Oct 13, 2026

Conversation

corona10 commented Oct 10, 2026 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Oct 10, 2026

Uh oh!

bedevere-bot commented Oct 10, 2026

Uh oh!

corona10 commented Oct 10, 2026

Uh oh!

read-the-docs-community Bot commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

bedevere-bot commented Oct 10, 2026

Uh oh!

corona10 commented Oct 10, 2026

Uh oh!

mpage commented Oct 11, 2026

Uh oh!

corona10 commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corona10 commented Oct 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corona10 commented Oct 11, 2026

Uh oh!

corona10 commented Oct 11, 2026

Uh oh!

corona10 commented Oct 11, 2026

Uh oh!

methane commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mpage commented Oct 12, 2026

Uh oh!

corona10 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corona10 commented Oct 13, 2026

Uh oh!

methane Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

corona10 Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

corona10 Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

corona10 commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

corona10 commented Oct 13, 2026

Uh oh!

corona10 Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Oct 13, 2026

Uh oh!

mpage commented Oct 13, 2026

Uh oh!

mpage commented Oct 13, 2026

Uh oh!

corona10 commented Oct 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

corona10 commented Oct 10, 2026 •

edited by bedevere-app Bot

Loading

read-the-docs-community Bot commented Oct 10, 2026 •

edited

Loading

corona10 commented Oct 11, 2026 •

edited

Loading

corona10 commented Oct 11, 2026 •

edited

Loading

methane commented Oct 12, 2026 •

edited

Loading

corona10 commented Oct 12, 2026 •

edited

Loading

corona10 Oct 13, 2026 •

edited

Loading

corona10 Oct 13, 2026 •

edited

Loading

corona10 commented Oct 13, 2026 •

edited

Loading