Migrate pylibseekdb to nanobind stable ABI wheels (closes #8) - #37
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the pylibseekdb Python extension from pybind11 to nanobind and updates the packaging/CI configuration to target stable-ABI (abi3) wheels for Python 3.12+, while also extending the manylinux seekdb build helper to support aarch64.
Changes:
- Replace pybind11 bindings with nanobind (
NB_MODULE, nanobind CMake integration, updated CI deps). - Update Python wheel build configuration toward a reduced matrix and abi3 targeting for 3.12+.
- Add
SEEKDB_ARCH=aarch64support to the glibc 2.28 seekdb build script and document it.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/build-seekdb-glibc228.sh | Adds SEEKDB_ARCH selection to build seekdb inside the correct manylinux_2_28 image (x86_64/aarch64). |
| README.md | Updates top-level docs to reflect nanobind + new wheel matrix and aarch64 build instructions. |
| python/src/bindings.cpp | Ports bindings to nanobind and adjusts result row construction and module definitions. |
| python/README.md | Updates Python package platform requirements to include Linux aarch64. |
| python/pyproject.toml | Updates build-system deps and cibuildwheel/scikit-build settings for the new build strategy. |
| python/CMakeLists.txt | Switches CMake integration from pybind11 to nanobind and enables STABLE_ABI module build. |
| .github/workflows/pr-ci.yml | Updates CI Python build dependencies from pybind11 to nanobind. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
15
to
36
| # SEEKDB_ARCH: x86_64 (default) or aarch64 — selects the manylinux_2_28 image. | ||
| ARCH="${SEEKDB_ARCH:-x86_64}" | ||
| BUILD_TYPE="${1:-release}" | ||
|
|
||
| case "$ARCH" in | ||
| x86_64) | ||
| IMAGE="${SEEKDB_EL8_IMAGE:-quay.io/pypa/manylinux_2_28_x86_64:2026.03.20-1}" | ||
| ;; | ||
| aarch64) | ||
| IMAGE="${SEEKDB_EL8_IMAGE:-quay.io/pypa/manylinux_2_28_aarch64:2026.03.20-1}" | ||
| ;; | ||
| *) | ||
| echo "error: unsupported SEEKDB_ARCH=$ARCH (use x86_64 or aarch64)" >&2 | ||
| exit 1 | ||
| ;; | ||
| esac | ||
|
|
||
| echo "=== building seekdb ($BUILD_TYPE) for $ARCH in $IMAGE ===" | ||
|
|
||
| docker run --rm \ | ||
| -v "$REPO":/seekdb -w /seekdb \ | ||
| -e BUILD_TYPE="$BUILD_TYPE" \ |
Comment on lines
+163
to
170
| Py_ssize_t n = static_cast<Py_ssize_t>(cells.size()); | ||
| nb::tuple row = nb::steal<nb::tuple>(PyTuple_New(n)); | ||
| if (!row.is_valid()) | ||
| throw std::runtime_error("failed to allocate tuple"); | ||
| for (Py_ssize_t i = 0; i < n; ++i) { | ||
| if (PyTuple_SetItem(row.ptr(), i, cells[i].release().ptr()) < 0) | ||
| throw nb::python_error(); | ||
| } |
Comment on lines
+164
to
+168
| nb::tuple row = nb::steal<nb::tuple>(PyTuple_New(n)); | ||
| if (!row.is_valid()) | ||
| throw std::runtime_error("failed to allocate tuple"); | ||
| for (Py_ssize_t i = 0; i < n; ++i) | ||
| PyTuple_SET_ITEM(row.ptr(), i, cells[i].release().ptr()); |
Comment on lines
+15
to
20
| IMAGE="${SEEKDB_EL8_IMAGE:-quay.io/pypa/manylinux_2_28:2026.03.20-1}" | ||
| BUILD_TYPE="${1:-release}" | ||
|
|
||
| echo "=== building seekdb ($BUILD_TYPE) in $IMAGE ===" | ||
|
|
||
| docker run --rm \ |
Use cp311 per-platform wheels plus cp312-abi3 for Python 3.12+, cutting the release matrix from 12 to 6 wheels and adding linux aarch64 support. Co-authored-by: Cursor <cursoragent@cursor.com>
…ion. Point minimum-version at build-system.requires and require scikit-build-core>=0.10 so pip isolation and tool config stay aligned. Co-authored-by: Cursor <cursoragent@cursor.com>
Clarify that scikit-build-core only applies abi3 tagging when the build Python is >= the configured py-api version. Co-authored-by: Cursor <cursoragent@cursor.com>
Collect column values in std::vector<nb::object> for exception safety, then steal into a single PyTuple without an intermediate list. Co-authored-by: Cursor <cursoragent@cursor.com>
Restore nb::call_guard<nb::gil_scoped_release> on begin, commit, rollback, and execute so other Python threads can run during I/O waits. Co-authored-by: Cursor <cursoragent@cursor.com>
Connections are only created via connect() as shared_ptr, which lets cursor()'s shared_from_this() work without pybind11-style holders. Co-authored-by: Cursor <cursoragent@cursor.com>
Use the multi-arch manylinux_2_28 image without SEEKDB_ARCH, fill new tuples with PyTuple_SET_ITEM, and bump macOS arm64 minimum to 15.6. Co-authored-by: Cursor <cursoragent@cursor.com>
hnwyllmm
force-pushed
the
codex/investigate-issue
branch
from
July 10, 2026 06:42
7ec4993 to
ab8bf82
Compare
Comment on lines
+166
to
+170
| nb::tuple row = nb::steal<nb::tuple>(PyTuple_New(n)); | ||
| if (!row.is_valid()) | ||
| throw std::runtime_error("failed to allocate tuple"); | ||
| for (Py_ssize_t i = 0; i < n; ++i) | ||
| PyTuple_SET_ITEM(row.ptr(), i, cells[i].release().ptr()); |
| repair-wheel-command = "auditwheel repair --plat {plat} -w {dest_dir} {wheel}" | ||
|
|
||
| [tool.cibuildwheel.macos] | ||
| archs = ["arm64"] |
Comment on lines
+44
to
+45
| - Linux x86_64 or aarch64 with glibc >= 2.28 (Alpine / musl not supported yet) | ||
| - macOS arm64 >= 15.6 |
Comment on lines
+15
to
+19
| IMAGE="${SEEKDB_EL8_IMAGE:-quay.io/pypa/manylinux_2_28:2026.03.20-1}" | ||
| BUILD_TYPE="${1:-release}" | ||
|
|
||
| echo "=== building seekdb ($BUILD_TYPE) in $IMAGE ===" | ||
|
|
Member
Author
There was a problem hiding this comment.
Good catch — the PR description was stale. We intentionally removed SEEKDB_ARCH and now use the multi-arch manylinux_2_28 image so Docker pulls the host-matching architecture (x86_64 or aarch64). Cross-arch builds via QEMU/--platform are out of scope; each linux wheel arch is built on a matching host/CI runner.
Updated the PR description and README accordingly in ab8bf82.
Keep chained .def() calls on the same line as connection_class. Co-authored-by: Cursor <cursoragent@cursor.com>
Comment on lines
+169
to
+170
| for (Py_ssize_t i = 0; i < n; ++i) | ||
| PyTuple_SET_ITEM(row.ptr(), i, cells[i].release().ptr()); |
PyTuple_SET_ITEM is unavailable under Py_LIMITED_API; PyTuple_SetItem is the stable-ABI-safe way to fill a newly allocated tuple. CI now exercises the cp312+ limited API build path. Co-authored-by: Cursor <cursoragent@cursor.com>
Comment on lines
+165
to
172
| Py_ssize_t n = static_cast<Py_ssize_t>(cells.size()); | ||
| nb::tuple row = nb::steal<nb::tuple>(PyTuple_New(n)); | ||
| if (!row.is_valid()) | ||
| throw std::runtime_error("failed to allocate tuple"); | ||
| for (Py_ssize_t i = 0; i < n; ++i) { | ||
| if (PyTuple_SetItem(row.ptr(), i, cells[i].release().ptr()) < 0) | ||
| throw nb::python_error(); | ||
| } |
Member
Author
There was a problem hiding this comment.
Fixed in the latest commit: PyTuple_New failure now raises nb::python_error() so the original Python exception (e.g. MemoryError) is preserved.
On PyTuple_SetItem, CPython steals the reference even when it returns -1 (it decrefs newitem on error paths), so an extra Py_DECREF on failure would double-free. The loop now assigns item before calling PyTuple_SetItem for clarity; with a freshly allocated tuple and in-range indices, failure is not expected in practice.
Comment on lines
201
to
+205
| - &setup-python | ||
| name: Setup Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' | ||
| python-version: '3.12' |
Run hybrid-search integration checks after each cibuildwheel build and provide scripts/verify-wheel.sh for manual verification of wheelhouse output. Co-authored-by: Cursor <cursoragent@cursor.com>
Comment on lines
45
to
+48
| | `make pylibseekdb` | `pylibseekdb.cpython-*.so` (Python extension) | | ||
| | `make seekdb_tests` | `test_one_client_process`, `test_two_clients_threads` | | ||
| | `make wheel` | `wheelhouse/pylibseekdb-*.whl` via cibuildwheel | | ||
| | `make wheel` | `wheelhouse/pylibseekdb-*.whl` via cibuildwheel (runs `seekdb_test.py` per wheel) | | ||
| | `make verify-wheel` | Re-run `seekdb_test.py` against wheels in `build/wheelhouse/` | |
Comment on lines
+38
to
+50
| local tmp | ||
| tmp="$(mktemp -d)" | ||
|
|
||
| "$py" -m venv "$tmp/venv" | ||
| # shellcheck disable=SC1091 | ||
| source "$tmp/venv/bin/activate" | ||
| python -m pip install -q --upgrade pip | ||
| python -m pip install -q "$wheel" | ||
|
|
||
| mkdir -p "$tmp/work" | ||
| (cd "$tmp/work" && python "$TEST_PY") | ||
| rm -rf "$tmp" | ||
| } |
Comment on lines
+47
to
+49
| sql = '''SELECT json_pretty(DBMS_HYBRID_SEARCH.SEARCH('doc_table', @parm))''' | ||
| cursor.execute(sql) | ||
| print(cursor.fetchall()) |
Comment on lines
+1
to
+6
| # -*- coding: utf-8 -*- | ||
| import pylibseekdb as seekdb | ||
|
|
||
| seekdb.open() | ||
| conn = seekdb.connect("test") | ||
| cursor = conn.cursor() |
Throw nb::python_error after PyTuple_New failure instead of masking MemoryError with std::runtime_error. Co-authored-by: Cursor <cursoragent@cursor.com>
Comment on lines
+12
to
+29
| pick_python() { | ||
| local wheel="$1" | ||
| if [[ "$wheel" == *cp311-* ]]; then | ||
| if command -v python3.11 >/dev/null 2>&1; then | ||
| command -v python3.11 | ||
| return | ||
| fi | ||
| fi | ||
| if [[ "$wheel" == *abi3* ]] || [[ "$wheel" == *cp312-* ]]; then | ||
| for py in python3.14 python3.13 python3.12 python3; do | ||
| if command -v "$py" >/dev/null 2>&1; then | ||
| echo "$py" | ||
| return | ||
| fi | ||
| done | ||
| fi | ||
| command -v python3 | ||
| } |
Comment on lines
+31
to
+50
| verify_one_wheel() { | ||
| local wheel="$1" | ||
| local py | ||
| py="$(pick_python "$wheel")" | ||
|
|
||
| echo "=== verifying wheel: $wheel (python: $py) ===" | ||
|
|
||
| local tmp | ||
| tmp="$(mktemp -d)" | ||
|
|
||
| "$py" -m venv "$tmp/venv" | ||
| # shellcheck disable=SC1091 | ||
| source "$tmp/venv/bin/activate" | ||
| python -m pip install -q --upgrade pip | ||
| python -m pip install -q "$wheel" | ||
|
|
||
| mkdir -p "$tmp/work" | ||
| (cd "$tmp/work" && python "$TEST_PY") | ||
| rm -rf "$tmp" | ||
| } |
Comment on lines
+17
to
+23
| sql = '''insert into doc_table values | ||
| (1, '[1,2,3]', "hello world", "oceanbase Elasticsearch database"), | ||
| (2, '[1,2,1]', "hello world, what is your name", "oceanbase mysql database"), | ||
| (3, '[1,1,1]', "hello world, how are you", "oceanbase oracle database"), | ||
| (4, '[1,3,1]', "real world, where are you from", "postgres oracle database"), | ||
| (5, '[1,3,2]', "real world, how old are you", "redis oracle database"), | ||
| (6, '[2,1,1]', "hello world, where are you from", "starrocks oceanbase database")''' |
Comment on lines
201
to
+205
| - &setup-python | ||
| name: Setup Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.11' | ||
| python-version: '3.12' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
STABLE_ABIfor Python 3.12+ wheels.cp311-cp311for Python 3.11 pluscp312-abi3for 3.12+ on linux x86_64, linux aarch64, and macOS arm64.pyproject.toml, build docs/CI, and simplifybuild-seekdb-glibc228.shto use the multi-archmanylinux_2_28image (Docker selects the host architecture automatically).Test plan
cmake --build build --target pylibseekdbsucceedspip wheelproducescp312-abi3wheel on Python 3.13pylibseekdbsuccessfullycibuildwheel --print-build-identifiersreports 6 expected buildsmake wheel/ cibuildwheel run on linux with architecture-matchingseekdbbinariesMade with Cursor