[C++][Python] SIGSEGV in bundled mimalloc mi_thread_init when libarrow is first loaded on a non-main thread that exits (mimalloc 3.3.x, pyarrow 25.0.0)

[L3o-pold]

Describe the bug, including details regarding any error messages, version, and platform

pyarrow 25.0.0 segfaults deterministically when pyarrow is first imported on a non-main thread that later exits, and a different thread then performs its first Arrow allocation. The crash is inside the bundled mimalloc's thread initialization:

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)  KERN_INVALID_ADDRESS at 0x0000000000000018

libarrow.2500.dylib   mi_heap_main
libarrow.2500.dylib   mi_thread_init
libarrow.2500.dylib   _mi_malloc_generic
libarrow.2500.dylib   mi_theap_malloc_zero_aligned_at_generic
libarrow.2500.dylib   arrow::(anonymous namespace)::MimallocAllocator::AllocateAligned(long long, long long, unsigned char**)
libarrow.2500.dylib   arrow::BaseMemoryPoolImpl<arrow::(anonymous namespace)::MimallocAllocator>::Allocate(...)
libarrow.2500.dylib   arrow::PoolBuffer::Reserve(long long)
libarrow.2500.dylib   arrow::PoolBuffer::Resize(long long, bool)
libarrow_python.2500.dylib  arrow::py::ConvertPySequence / arrow::py::NdarrayToArrow

Minimal reproduction

import threading

def imp():
    import pyarrow  # dlopens libarrow -> bundled mimalloc initializes on this thread

def alloc():
    import pyarrow as pa
    pa.array(["x"] * 5000)

a = threading.Thread(target=imp); a.start(); a.join()
b = threading.Thread(target=alloc); b.start(); b.join()

Result: SIGSEGV (exit code 139). The same script:

• does not crash if import pyarrow happens on the main thread first;
• does not crash with ARROW_DEFAULT_MEMORY_POOL=system;
• does not crash on pyarrow 24.0.0.

Root cause

This is upstream mimalloc bug microsoft/mimalloc#1287 ("mimalloc >= 3.3.0 causes segmentation faults when used from multiple threads", closed 2026-06-20): mimalloc treats the first thread that initializes it as the process main thread; when that thread exits, subsequent threads crash in mi_thread_init/_mi_malloc_generic. mimalloc 3.2.x is unaffected.

Arrow 25.0.0 is the first release affected because GH-49772 / PR #49801 bumped ARROW_MIMALLOC_BUILD_VERSION from v3.2.7 to v3.3.1 (confirmed in cpp/thirdparty/versions.txt at tag apache-arrow-25.0.0).

The upstream fix exists only on mimalloc's dev3 branch (commit microsoft/mimalloc@b92c116b67d0, "allow initial main thread to terminate before the process terminates (see issue microsoft/mimalloc#1287)"); the latest tagged mimalloc release (v3.3.2, 2026-04-29) predates it, so there is currently no tagged mimalloc version Arrow could bump to.

Real-world impact

This import pattern is not exotic — it is structurally guaranteed by Streamlit + pandas 3:

• pandas 3.x imports pyarrow eagerly (default pyarrow-backed str dtype);
• Streamlit executes user scripts (including their imports) on short-lived ScriptRunner threads, never on the main thread.

So in any Streamlit app using pandas 3 + pyarrow 25, libarrow first loads on script-run thread #1, and the next script run (a simple page reload) segfaults the whole server on its first string-column conversion (pandas maybe_convert_objects → arrow::py::NdarrayToArrow). We observed 7+ identical crashes per day on one such app before diagnosing.

Workarounds

• ARROW_DEFAULT_MEMORY_POOL=system (verified);
• import pyarrow on the main thread before any worker thread touches it (verified);
• pin pyarrow<=24.0.0 (bundles unaffected mimalloc v3.2.7).

Suggested resolution

Consider reverting the bundled mimalloc to the v3.2.x line until a tagged mimalloc release contains the #1287 fix, or picking up that fix as a patch on top of v3.3.1.

Related but distinct: #50428 (interpreter-teardown crash when co-loading another mimalloc-v3-bundling extension; the ARROW_DEFAULT_MEMORY_POOL=system workaround reportedly does not help there, whereas it does here).

Environment

• pyarrow 25.0.0 (PyPI wheel), pandas 3.0.3, streamlit 1.59.1
• CPython 3.14.5 (Homebrew), macOS 26 (Darwin 25.5.0), Apple Silicon
• Note: mimalloc#1287 was reproduced upstream on Linux and Windows as well, so this is unlikely to be macOS-specific.

Component(s)

C++, Python

[lukasmasuch]

Streamlit dev here 👋 @L3o-pold, thanks for investigating!

The PyArrow 25.0.0 update broke our CI and, very likely, also a significant number of apps that get updated to this version. Is there a way to get this patched soon? cc @raulcd

[raulcd]

I haven't investigated yet so I don't fully understand the issue or the solution but from my initial understanding we could add to our ci/vcpkg/ports.patch the patch from the upstream commit microsoft/mimalloc@b92c116b67d0 and build the wheels with a patched mimalloc. There were some discussions about a possible 25.0.1 patch release but it's not entirely clear when those patches will be ready.

@kou @pitrou any thoughts on the issue?

[kou]

I don't object that we backport the patch by ourselves but I'm negative for the approach because it may introduce another problem.

How about using system malloc() on non-main thread something like the following?

diff --git a/cpp/src/arrow/memory_pool.cc b/cpp/src/arrow/memory_pool.cc
index 0b4843bec3..d00760291d 100644
--- a/cpp/src/arrow/memory_pool.cc
+++ b/cpp/src/arrow/memory_pool.cc
@@ -130,12 +130,40 @@ std::optional<MemoryPoolBackend> UserSelectedBackend() {
   return user_selected_backend;
 }
 
+bool IsMainThread() {
+#ifdef __linux__
+  // The main thread's ID must equal to process ID on Linux.
+  return gettid() == getpid();
+#elif defined(__APPLE__) || defined(BSD)
+  // We can use pthread_main_np() on BSD family.
+  return pthread_main_np() == 1;
+#elif defined(_WIN32)
+  // Windows doesn't provide an API for this. We can implement a
+  // heuristic approach.
+  return true;
+#else
+  // Assume the current thread is the main thread on others.
+  return true;
+#endif
+}
+
 MemoryPoolBackend DefaultBackend() {
   auto backend = UserSelectedBackend();
   if (backend.has_value()) {
     return backend.value();
   }
-  struct SupportedBackend default_backend = SupportedBackends().front();
+  static struct SupportedBackend default_backend;
+  static std::once_flag once;
+  std::call_once(once, [&]() {
+    if (IsMainThread()) {
+      default_backend = SupportedBackends().front();
+    } else {
+      // Fallback to the system backend.
+      // We can remove this once mimalloc()'s thread related problem is fixed.
+      // See also: https://github.com/microsoft/mimalloc/issues/1287
+      default_backend = SupportedBackends().back();
+    }
+  });
   return default_backend.backend;
 }