Rename mock TDS Python package + add sandbox publish pipeline (test-only)#80
Rename mock TDS Python package + add sandbox publish pipeline (test-only)#80saurabh500 wants to merge 14 commits into
Conversation
Change the PyPI distribution name and Python import module from mssql-mock-tds-py / mssql_mock_tds_py to mssql-mock-tds / mssql_mock_tds. The folder and Cargo package name stay mssql-mock-tds-py to avoid colliding with the existing mssql-mock-tds crate it depends on. Use a #[pyo3(name)] override so the pymodule fn keeps a distinct identifier from the dependency crate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
📊 Code Coverage Report
Diff CoverageDiff: main...HEAD, staged and unstaged changesNo lines with coverage information in this diff. 🔗 Quick Links |
* Add sandbox OneBranch pipeline to publish mock TDS artifacts Adds .pipeline/OneBranch/PublishFeeds-Sandbox.yml, a manual-only, test-only pipeline that builds and (opt-in) publishes the mock TDS Python wheels and crate to the mssql-rs_Public Azure Artifacts feed. Publishing is gated behind publishPython/publishCrate parameters, both default false (build + dry run only). Adds a single-abi3-wheel build script for the Linux container build. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extend sandbox pipeline with ARM64 and macOS wheel jobs Adds Linux ARM64, Windows ARM64, and macOS universal2 mock wheel build jobs to the sandbox PublishMockPython stage and wires them into the UploadPython dependsOn. macOS builds with a new vendored-openssl passthrough feature on mssql-mock-tds-py so the universal2 wheel statically links OpenSSL instead of Homebrew's libssl. Linux jobs keep system libssl (auditwheel skip); Windows uses SChannel. Still sandbox/test-only and opt-in. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix OneBranch template resolution in sandbox pipeline The sandbox pipeline inlines its stages under extends.parameters.stages while extending the OneBranch governed template, so unqualified template: includes resolved against the GovernedTemplates repo and failed (e.g. cargo-authenticate-template.yml not found). Suffix every internal /.pipeline/... template include with @self so they resolve against this repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Conform sandbox artifact names to OneBranch convention The governed OneBranch job template requires PublishPipelineArtifact names to be drop_<Stage>_<Job>_<ob_artifactSuffix>. Rename the five mock wheel artifacts accordingly so stage validation passes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stamp sandbox wheel version as SemVer prerelease cargo metadata rejects the PEP 440 dotted '.dev' suffix in Cargo.toml ('unexpected character . after patch version number'), failing maturin before the build. Use the SemVer-valid '-dev' form for both pyproject.toml and Cargo.toml; PEP 440 normalizes it back to '.dev' for the wheel, and the matching versions satisfy maturin's pyproject/Cargo consistency check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix sandbox macOS PEP 668 install and inline crate version stamping macOS wheel build failed at 'pip install --user pipx' with externally-managed-environment (PEP 668) on the Homebrew Python. Bootstrap pipx with --break-system-packages only when missing, and install the wheel package the same way for the re-tag step. cargo publish --dry-run failed because the shared stamp-crate-versions.ps1 uses a global ^version regex that also rewrote the [dependencies.uuid] version field to the prerelease version, corrupting the uuid requirement. Stamp inline instead, replacing only the first ^version line (the [package] version) in mssql-tds and mssql-mock-tds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix macOS sandbox wheel version stamp for BSD sed The macOS Stamp step used the GNU-only empty-regex reuse form s//.../ which BSD sed rejects with "first RE may not be empty". Replace it with an explicit substitution pattern. Each mock manifest has exactly one line-starting version = (the package version), so a plain substitution stamps only that line. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Invoke wheel as a module in macOS re-tag step The --user/Homebrew wheel install does not place the wheel console script on PATH, so wheel tags failed with exit 127 (command not found). Call it as python3 -m wheel tags instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use UsePythonVersion for macOS mock wheel job Pin Python 3.12 via UsePythonVersion@0 instead of relying on the agent's externally-managed Homebrew python3. The selected interpreter is managed, writable, and on PATH, so the PEP 668 --break-system-packages workarounds and the off-PATH wheel/maturin console-script issues go away. Install maturin and wheel with python -m pip against that interpreter and re-tag with python -m wheel. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Run mock wheel upload job on the custom Windows pool The UploadPython job used the default governed Windows pool, whose build container has an empty Python tools cache and no outbound network, so UsePythonVersion@0 failed trying to download Python 3.12 (WinError 10013). Use the same custom pool/image as the Windows build jobs (RUST-1ES-POOL-WUS3 / RUST-Win22-Sql25-1P), which ships Python 3.12 and can pip install twine. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop unsupported --skip-existing from twine upload Azure Artifacts' PyPI endpoint rejects twine's --skip-existing flag (UnsupportedConfiguration). Each run stamps a unique .dev<date><BuildId> wheel version, so there is no existing version to skip. Remove the flag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build self-contained musllinux_1_2 wheels (Alpine) for both architectures using the repo's musllinux_*_rust container images. musl wheels statically vendor OpenSSL (MATURIN_FEATURES=vendored-openssl) so they don't depend on Alpine's libssl at the consumer side; the container build script now honors an optional MATURIN_FEATURES env. Wire both jobs into UploadPython. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the releaseVersion parameter is set, the seven Python wheel build jobs stamp the base version from pyproject.toml as-is (e.g. 1.0.0) instead of appending the .dev<date><buildId> prerelease suffix. Default remains the .dev prerelease so existing behavior is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a PreflightVersionCheck job that gates all seven wheel builds. When publishing a release (releaseVersion + publishPython both true), it authenticates pip to the feed and queries the PEP 503 simple index for mssql-mock-tds; if the base version from pyproject.toml is already published, the pipeline fails in seconds instead of building seven wheels and only hitting the duplicate-version rejection at twine upload. The check is best-effort on lookup failure (warns, continues) since the upload-time rejection remains the authoritative guard. For dev/dry-run runs the job is a no-op. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The mssql-rs_Public feed allows anonymous reads, so drop PipAuthenticate from the Python preflight and pass the public simple-index URL directly. Extend releaseVersion to the crate stage: when set, mssql-tds and mssql-mock-tds publish their base version (e.g. 1.0.0) instead of the -dev.<date>.<buildId> prerelease. Add a PreflightVersionCheck-style guard as the first step of PublishCrate that queries the public Cargo sparse index and fails fast if either crate version is already published, before cargo-authenticate and the rust install. Best-effort on lookup failure; cargo's duplicate-version rejection remains the authoritative guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…thon-pkg-ado-feed
…thon-pkg-ado-feed
There was a problem hiding this comment.
Pull request overview
This draft PR renames the mock TDS Python distribution/import to align on mssql-mock-tds / mssql_mock_tds, and adds a manual “sandbox” OneBranch pipeline intended to build (and optionally publish) mock test artifacts to an Azure Artifacts feed.
Changes:
- Rename the mock Python distribution/module name (
mssql-mock-tds-py→mssql-mock-tds, importmssql_mock_tds_py→mssql_mock_tds) and update one rs-only test accordingly. - Add a sandbox OneBranch pipeline to build multi-target abi3 wheels and optionally publish wheels/crates with preflight duplicate-version checks.
- Add supporting scripts for container wheel builds and feed preflight version checks, plus a
vendored-opensslpassthrough feature for portable wheels.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
scripts/build-mock-python-wheel-in-container.sh |
Container build helper for a single abi3 mock wheel (optionally with features). |
mssql-py-core/tests/rs-only-tests/test_mock_server_fedauth.py |
Updates mock bindings import and usage to mssql_mock_tds. |
mssql-mock-tds-py/src/lib.rs |
Renames the exported PyO3 module to mssql_mock_tds while avoiding symbol conflicts. |
mssql-mock-tds-py/pyproject.toml |
Renames the PyPI dist and sets maturin module-name. |
mssql-mock-tds-py/Cargo.toml |
Adds vendored-openssl passthrough feature for wheel portability scenarios. |
.pipeline/scripts/check-version-not-published.py |
Adds Python preflight to fail fast if a release version already exists in the feed. |
.pipeline/scripts/check-crate-version-not-published.ps1 |
Adds Cargo sparse-index preflight to detect already-published crate versions. |
.pipeline/OneBranch/PublishFeeds-Sandbox.yml |
Adds the manual-only sandbox build/publish pipeline for wheels + crates. |
The mock TDS Python module was renamed to mssql_mock_tds; the stale mssql_mock_tds_py import made the FedAuth-over-mock-server tests always ImportError and skip. Update the import, skip reason, and usage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move all step logic out of the sandbox pipeline YAML into reusable .pipeline/scripts helpers, de-duplicating the repeated version-stamp and container-build blocks (bash x5/x4, pwsh x2/x2). The YAML now only wires jobs, pools and parameters to scripts. Wheel dev versions emit .dev directly (PEP 440) instead of relying on -dev normalization. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 22949241-dd2a-4141-bd79-034d18c14e82
…thon-pkg-ado-feed
There was a problem hiding this comment.
Superseded — see the updated review with inline comments.
David-Engel
left a comment
There was a problem hiding this comment.
Summary
Two changes: (1) renames the mock TDS Python distribution/import from mssql-mock-tds-py/mssql_mock_tds_py to mssql-mock-tds/mssql_mock_tds (folder and Cargo package names intentionally unchanged), and (2) adds a manual-only, opt-in OneBranch "sandbox" pipeline plus helper scripts to build multi-target abi3 wheels and the mock crates, and optionally publish them to mssql-rs_Public. Overall this is well-structured, safe-by-default (dry-run unless opted in), and the rename is internally consistent. No blocking issues. Specific suggestions and nits are left as inline comments.
Verified as correct
- Safe defaults:
trigger/pr: none, publish is opt-in,UploadPythongated by bothsucceeded()/ne(Build.Reason,'PullRequest')and${{ if eq(publishPython, true) }}. - No hardcoded secrets; feed tokens come from pipeline secret variables via
env:. #[pyo3(name = "mssql_mock_tds")]+[tool.maturin] module-name+ pyproject dist name are mutually consistent; the fn was renamed tomssql_mock_tds_moduleto avoid clashing with themssql_mock_tdscrate import.- Dependency-order crate publish (
mssql-tdsbeforemssql-mock-tds) withpin-mssql-tds-dep-version.ps1adding the requiredversionto the path dep — correct forcargo publish. - Preflight checks are best-effort (warn + exit 0 on lookup failure) with
twine/cargoas the authoritative duplicate guard — reasonable. - Rename is consistent — the only remaining
mssql_mock_tds_py/mssql-mock-tds-pyreferences (folder path, Cargo package name, skipifreasonstrings) are intentionally preserved;dev/test-python.shonly builds viamaturin developand needs no change.
|
|
||
| case "${RELEASE_VERSION:-}" in | ||
| True|true|TRUE) VER="${BASE}" ;; # release: publish BASE as-is | ||
| *) VER="${BASE}.dev$(date -u +%Y%m%d)${BUILD_BUILDID:-}" ;; # PEP 440 dev release segment (.devN) |
There was a problem hiding this comment.
Suggestion (dev-version date skew): each of the 7 parallel build jobs runs this script independently, so date -u +%Y%m%d is evaluated per-job. $BUILD_BUILDID is constant, but if a publishPython=true, releaseVersion=false run straddles UTC midnight the wheels for different platforms get different .devN versions, and the feed ends up with two versions each missing some platform tags. Consider computing the version once (e.g. in PreflightVersionCheck or a small setup job) and passing it to all build jobs via a pipeline output variable. Same root cause applies to the PowerShell stamper. Low probability, but a genuine inconsistency.
There was a problem hiding this comment.
Addressed in 52a0bf8. The wheel version is now computed once in the PreflightVersionCheck job (compute-mock-wheel-version.ps1, UTC date) and published as the computeVersion.mockWheelVersion output variable. All 7 build jobs map that single value and pass it to the stamper as WHEEL_VERSION / -Version, so the whole run shares one version even across a UTC-midnight boundary. Stampers fall back to self-computing only if the variable is empty.
| } | ||
| else { | ||
| $dev = "$(Get-Date -Format 'yyyyMMdd')$BuildId" | ||
| $ver = "$base.dev$dev" # PEP 440 dev release segment (.devN) for the wheel |
There was a problem hiding this comment.
Suggestion (dev-version date skew, cont.): same issue as the .sh stamper — Get-Date is evaluated independently in each parallel build job, so a run spanning UTC midnight can yield mismatched .devN versions across platforms. Computing the version once and passing it as a pipeline variable to all build jobs would keep the wheel set consistent.
There was a problem hiding this comment.
Addressed in 52a0bf8 along with the .sh stamper: the version is resolved once upstream and passed in via -Version '$(mockWheelVersion)', so Get-Date is no longer evaluated per-job. The local-vs-UTC mismatch between the two stampers is gone too since the single compute step uses UTC.
| set -euo pipefail | ||
|
|
||
| : "${OB_OUTPUTDIRECTORY:?OB_OUTPUTDIRECTORY is required}" | ||
| export MACOSX_DEPLOYMENT_TARGET=15.0 |
There was a problem hiding this comment.
Suggestion (macOS deployment target): MACOSX_DEPLOYMENT_TARGET=15.0 (paired with the macosx_15_0_universal2 retag) is built on the macOS-14 image, so it targets an OS newer than the build SDK and restricts the wheel to macOS 15+. Unless macOS 15 is specifically required, a lower target (e.g. 11.0) gives much broader install compatibility. Low priority for a test-only artifact, but worth a conscious choice.
There was a problem hiding this comment.
Addressed in 52a0bf8. Lowered MACOSX_DEPLOYMENT_TARGET to 11.0 and retag to macosx_11_0_universal2 for broad install compatibility.
| # mssql-tds must be queryable in the feed index before mssql-mock-tds is | ||
| # verified/published. Sparse index propagation is usually quick, but allow a short | ||
| # settle window for the sandbox. | ||
| Start-Sleep -Seconds 30 |
There was a problem hiding this comment.
Suggestion (fixed sleep is racy): a fixed Start-Sleep -Seconds 30 for sparse-index propagation can be both too short (dependent publish fails) and wastefully long. A poll-with-retry against the sparse index — reusing the index-path logic already in check-crate-version-not-published.ps1 — would be more robust. Acceptable for a sandbox.
There was a problem hiding this comment.
Addressed in 52a0bf8. Replaced the fixed Start-Sleep -Seconds 30 with a poll-with-retry (Wait-CrateOnIndex) against the sparse index for mssql-tds@<version>. The index-path logic is now a shared Get-CargoIndexPath.ps1 dot-sourced by both cargo-publish-mock.ps1 and check-crate-version-not-published.ps1. Falls back to a short sleep if the index URL/version aren't supplied.
| $ErrorActionPreference = 'Stop' | ||
|
|
||
| $date = Get-Date -Format 'yyyyMMdd' | ||
| $rx = [regex]'(?m)^(version\s*=\s*)"[^"]+"' |
There was a problem hiding this comment.
Suggestion (^version first-match assumption): the Replace(..., 1) count-limited substitution relies on [package].version being the first line-anchored version = "…" in each Cargo.toml. The comment explaining why the global regex was avoided is good, but this is still positional-fragile if a manifest is reordered so another table precedes [package]. A [package]-section-scoped edit (or a TOML-aware tool) would be safer. The same positional assumption exists in the wheel stampers; low risk given current file layout.
There was a problem hiding this comment.
Addressed in 52a0bf8. stamp-crate-versions-sandbox.ps1 now scopes the edit to the [package] table (isolates from the header to the next ^[ table or EOF, then replaces version inside), so table order no longer matters. Verified against a synthetic manifest with a [dependencies.uuid] version line preceding [package] — only the package version changes.
| target = normalize_version(base) | ||
| print(f"Release preflight: checking feed for {package}=={base} (normalized {target})") | ||
|
|
||
| import os |
There was a problem hiding this comment.
Nit: import os is buried mid-function inside main(). Move it up with the other top-level imports.
There was a problem hiding this comment.
Addressed in 52a0bf8 — moved import os up with the other top-level imports.
- Compute the mock wheel version once in PreflightVersionCheck and pass it to all build jobs via an output variable, so parallel wheels never get mismatched .devN versions across a UTC-midnight boundary. - Lower macOS deployment target 15.0 -> 11.0 for broader wheel compatibility. - Poll the sparse index for mssql-tds instead of a fixed 30s sleep before publishing mssql-mock-tds; extract shared Get-CargoIndexPath helper. - Scope the sandbox crate version stamp to the [package] table. - Move buried 'import os' to top-level imports in check-version-not-published.py Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: da91a448-51e4-40e5-8fdb-2b032719c2cf
Summary
Two related changes toward publishing the mock TDS test artifacts to the
mssql-rs_PublicAzure Artifacts feed, opened as a draft for early review:pip install mssql-mock-tds/import mssql_mock_tds(the-pysuffix was awkward).Part 1 — Package rename
What changed
pyproject.toml[project].name):mssql-mock-tds-py→mssql-mock-tds.[tool.maturin] module-name+#[pyo3(name = ...)]):mssql_mock_tds_py→mssql_mock_tds.mssql-py-core/tests/rs-only-tests/test_mock_server_fedauth.pyupdated to the new import name.What intentionally did not change (avoids conflicts)
mssql-mock-tds-py/— renaming would collide with the existingmssql-mock-tds/crate directory.[package].namemssql-mock-tds-py— the binding crate path-depends onmssql-mock-tds; two Cargo packages can't share a name.The PyPI name and the Cargo crate name
mssql-mock-tdslive in different feed protocols (PyPI vs Cargo), so they don't collide — they intentionally match.Implementation note: naming the
#[pymodule]fnmssql_mock_tdsclashed with themssql_mock_tdsdependency crate (use mssql_mock_tds::...). Resolved by keeping a distinct fn identifier (mssql_mock_tds_module) with a#[pyo3(name = "mssql_mock_tds")]override.Part 2 — Sandbox publish pipeline
By default nothing this pipeline produces is a production artifact — wheels are stamped with PEP 440
.devNprereleases and crates with-dev.<date>.<buildId>prereleases. Publishing is opt-in; with default parameters the pipeline only builds and runs a dry run. An opt-inreleaseVersiontoggle exists to publish clean base versions (see below).What's added
.pipeline/OneBranch/PublishFeeds-Sandbox.yml—name: "Publish Feeds (Sandbox)",trigger: none/pr: none, two independent stages:>=3.9) wheel for the mock package across 7 targets (one wheel per target, no per-Python loop), thentwine uploads to the feed when enabled.cargo publishesmssql-tdsthenmssql-mock-tds(dependency order) to the feed's Cargo index.scripts/build-mock-python-wheel-in-container.sh— singlematurin build --auditwheel skip --manifest-path mssql-mock-tds-py/Cargo.tomlfor the Linux container builds; honors an optionalMATURIN_FEATURESenv (used by the musllinux jobs to vendor OpenSSL)..pipeline/scripts/check-version-not-published.py— dependency-free preflight that queries the feed's public PyPI simple index and fails fast if the Python base version is already published..pipeline/scripts/check-crate-version-not-published.ps1— preflight that queries the feed's public Cargo sparse index and fails fast ifmssql-tdsormssql-mock-tdsalready has the target version.mssql-mock-tds-py/Cargo.toml— newvendored-opensslpassthrough feature (["mssql-mock-tds/vendored-openssl"]) used by the macOS and musllinux builds.Build matrix
RUST-1ES-POOL-WUS3/ manylinux_2_34 x86_64RUST-1ES-POOL-ARM-WUS3/ manylinux_2_34 aarch64RUST-1ES-POOL-WUS3/ musllinux_1_2 x86_64RUST-1ES-POOL-ARM-WUS3/ musllinux_1_2 aarch64RUST-1ES-POOL-WUS3/ RUST-Win22-Sql25-1PRUST-1ES-POOL-ARM-WUS3/ RUST-WINSRV-ARMParameters (all default
false= safe dry run)publishPythontwine uploadto feedpublishCratecargo publish --dry-runmssql-tdsthenmssql-mock-tdsreleaseVersion.devN/-dev.<date>.<buildId>)1.0.0), guarded by preflightRelease versioning & preflight conflict checks
releaseVersion=false(default), wheels and crates get prerelease suffixes so repeated sandbox runs never collide.releaseVersion=true, the Python wheel and both crates publish their base version as-is (e.g.1.0.0) — a real release.twine upload(PyPI) andcargo publish(Cargo), so a transient lookup blip never blocks a legitimate release.versioninmssql-mock-tds-py/pyproject.toml(Python) and/ormssql-tds/Cargo.toml(both crates share the version), then queue with the relevantpublish*flag andreleaseVersion=true. Forgetting to bump → preflight fails fast.How to run
Queue manually in Azure DevOps (no CI/PR trigger): Run pipeline → select this branch → leave all toggles off for a build + dry run, or flip
publishPython/publishCrate(+ optionallyreleaseVersion) to push artifacts.Notes / intentional choices
mssql-mock-tdswhile the Cargo package/folder staysmssql-mock-tds-py; the build always passes--manifest-path.template:includes are suffixed@selfso they resolve against this repo (not the OneBranch GovernedTemplates repo) when defined inline underextends.parameters.stages.libssl.so.3via--auditwheel skip(glibc 2.28 vendoring is out of scope). musllinux and macOS wheels statically vendor OpenSSL so they are self-contained.mssql-mock-tdspulls theopensslcrate oncfg(not(windows))to build the PKCS#12 identity (the TLS handshake itself uses the platform backend — SecureTransport on macOS), so the non-glibc targets build with--features vendored-opensslrather than dynamically linking the distro's libssl. musllinux uses the repo'smusllinux_1_2_*_rustcontainer images.setup-nodejs-path.ps1+install-nodejs-to-agent.ps1) before the ADO tasks, matchingstages.yml.mssql-tdsdep version, and publishesmssql-tdsfirst so the verify build can resolve it.Validation
cargo checkonmssql-mock-tds-pypasses (validated by temporarily isolating the crate from the parent workspace; reverted).yaml.safe_loadparsesPublishFeeds-Sandbox.yml;bash -npasses on the container build script;cargo metadata --features vendored-opensslresolves the passthrough feature graph.mssql-mock-tds-pyreferences are folder/crate-name references, which are intentionally preserved.Follow-ups (not in this PR)
PublishNightlyFeeds.yml.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com