Skip to content

[FEATURE] Run one CI job against a real SurrealDB so "the query returns plausible garbage" is catchable #153

Description

@RobertSigmundsson

Problem Statement

A suggestion rather than a defect report, and entirely yours to weigh — you know the cost of your
CI minutes and we do not.

CI never starts a SurrealDB instance, so every test that needs one skips. That leaves the only
production backend untested in the pipeline, and it is the reason #143 was necessary: the test
that would have caught those three bugs was added in the same PR that introduced them (#139,
tests/integration/test_surrealdb_pinning.py) and has been skipping ever since.

Measured on v3.0.3:

$ grep -rln "SURREALDB_URL" tests/ | wc -l
26
$ grep -ci "surreal" .github/workflows/ci.yml
1                      # and that one occurrence is `--cov=surreal_memory`
$ grep -rn "services:" .github/workflows/*.yml
                       # no output — no workflow starts any service

26 test files are gated on SURREALDB_URL. The test job installs .[dev,server] — not even
the surrealdb extra — and runs pytest tests/ -m "not stress", so all of them skip green.

The parity ratchet added around #139 catches "the method is missing". It cannot catch
"the method is present and returns something that looks right" — which is precisely what
#143 fixed: SELECT VALUE over an array field returned a list of lists, the unwrap heuristic
collapsed it, and the caller iterated a string character by character. No signature was wrong.
Only a live engine shows that.

Proposed Solution

One integration job with a SurrealDB service container. Everything else stays as it is.

Untested. We cannot run your CI, and we have not run this YAML anywhere — it is written
from the SurrealDB image's documented interface and the shape of your existing test job, not
from a green run. Treat it as a starting point, not a verified patch. The image tag, the
health-check invocation and the exact --user/--pass flags are the parts most likely to need
adjusting.

  integration:
    name: Integration (SurrealDB)
    runs-on: ubuntu-latest
    timeout-minutes: 15
    services:
      surrealdb:
        image: surrealdb/surrealdb:v3.2.0
        ports:
          - 8000:8000
        options: >-
          --user root
          --health-cmd "/surreal is-ready --endpoint http://localhost:8000"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 12
        env:
          SURREAL_USER: root
          SURREAL_PASS: root
          SURREAL_BIND: 0.0.0.0:8000
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: pip install -e ".[dev,server,surrealdb]"
      - name: Run the integration suite
        env:
          SURREALDB_URL: http://127.0.0.1:8000
          SURREALDB_USER: root
          SURREALDB_PASS: root
          SURREALDB_NS: ci
          SURREALDB_DB: ci
        run: pytest tests/integration -v --timeout=120 -m "not stress"

If the official image's entrypoint needs an explicit start command in the Actions service
context, the alternative is to run the same container as an ordinary step
(docker run -d -p 8000:8000 …) and poll /health before the test step — less elegant, but it
sidesteps every options: question above.

Use Case

# tests/integration/test_surrealdb_pinning.py:46-51 — today
pytestmark = [
    pytest.mark.integration,
    pytest.mark.skipif(
        not SURREALDB_URL, reason="requires SURREALDB_URL (live SurrealDB >= 3.2.0)"
    ),
]

With the job above, that marker stops being a permanent skip and starts being what it was meant
to be: a guard for local runs without a server.

Alternatives Considered

  • Keep it manual. That is the status quo, and #143 is what it costs.
  • Mock the engine. A mock reproduces the interface, which is the half that already works;
    it cannot reproduce SELECT VALUE returning a list of lists.
  • Only run it nightly. Cheaper, but a regression then lands on main first and is found
    later. Worth considering if the job proves slow.

Additional Context

Scope: the four files under tests/integration/ are enough to be worth it — the pinning module
alone went from 5 failed, 12 passed to green in #143, and that measurement only existed
because it was run by hand against a real server.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions