From be0fd70f84dbd8a3db8948ee450c46855a8174aa Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Sat, 1 Aug 2026 10:00:42 +0800 Subject: [PATCH 1/8] design --- bindings/README.md | 6 + bindings/docs/dynamic-extensions/README.md | 307 ++++++++++ .../docs/dynamic-extensions/alternatives.md | 333 +++++++++++ .../docs/dynamic-extensions/compatibility.md | 536 ++++++++++++++++++ bindings/docs/dynamic-extensions/nodejs.md | 384 +++++++++++++ bindings/docs/dynamic-extensions/python.md | 356 ++++++++++++ bindings/docs/dynamic-extensions/ruby.md | 331 +++++++++++ 7 files changed, 2253 insertions(+) create mode 100644 bindings/docs/dynamic-extensions/README.md create mode 100644 bindings/docs/dynamic-extensions/alternatives.md create mode 100644 bindings/docs/dynamic-extensions/compatibility.md create mode 100644 bindings/docs/dynamic-extensions/nodejs.md create mode 100644 bindings/docs/dynamic-extensions/python.md create mode 100644 bindings/docs/dynamic-extensions/ruby.md diff --git a/bindings/README.md b/bindings/README.md index dc448ce19c20..7f8d60266da9 100644 --- a/bindings/README.md +++ b/bindings/README.md @@ -35,6 +35,12 @@ For example, while the `opendal` crate might be at version `0.55.0`, a binding might be at version `0.47.0` or `0.49.2`. For updates and compatibility, use the specific binding version instead of the `opendal` crate version. +## Design Proposals + +* [Dynamic service and layer extensions](docs/dynamic-extensions/README.md) + explores independently installable packages for the Python, Ruby, and + Node.js bindings. The proposal is not implemented yet. + ## Getting Started Every binding should provide a `README.md` file to help users get started. diff --git a/bindings/docs/dynamic-extensions/README.md b/bindings/docs/dynamic-extensions/README.md new file mode 100644 index 000000000000..168274265c6b --- /dev/null +++ b/bindings/docs/dynamic-extensions/README.md @@ -0,0 +1,307 @@ + + +# Dynamic Service and Layer Extensions + +Status: pre-RFC design exploration. OpenDAL has not accepted or implemented the +extension interface described by these documents. + +This design allows a language binding to install services and layers as +independent packages. For example, an application can install only S3, +Timeout, and Foyer without rebuilding or replacing the base binding. + +The leading prototype candidate uses an exact-build shared native runtime. +Python, Ruby, and Node.js use different language interfaces over the same +native extension model. Selecting that candidate remains conditional on a +successful cross-platform packaging prototype and the OpenDAL RFC process. + +## Documents + +- [Design alternatives](alternatives.md) compares four possible extension + seams and explains why Design C is the leading prototype candidate. +- [Compatibility and ABI](compatibility.md) defines version identities, + loader checks, lifetime rules, and supported stability guarantees. +- [Python design](python.md) defines Python packages, discovery, typing, and + migration. +- [Ruby design](ruby.md) defines Ruby gems, blocking construction, and + middleware migration. +- [Node.js design](nodejs.md) defines npm packages, Node-API integration, + worker behavior, and ESM/CommonJS loading. + +These documents are preliminary design input. They are not an accepted RFC or +a commitment to package names, release dates, or compatibility guarantees. A +subsequent proposal must follow the [OpenDAL RFC +process](../../../core/core/src/docs/rfcs/README.md) before implementation. + +## Requirements + +The design must satisfy all of the following requirements: + +- A service or layer package can be installed without rebuilding the base + language binding. +- The base binding loads without dependencies required by unselected packages. + In particular, HDFS must not make Java, Hadoop, `libhdfs`, or `libjvm` a base + requirement. +- A service package owns its configuration schema, URI interpretation, + credentials, and redaction behavior. +- A layer package preserves the complete native `Layer` behavior, including + both service and operation-context composition. +- Stateful layers can be reused intentionally. Reusing one Throttle handle + shares one limiter; reusing one Foyer handle shares one cache. +- Layer construction can be asynchronous. Foyer must not force asynchronous + resource creation into a synchronous registration callback. +- The registry supports at least 1,000 installed manifests without loading + every native library during base import. +- Official and third-party packages use the same extension interface and + compatibility checks. +- The loader rejects incompatible native code before exchanging Rust values or + storing callbacks. +- Active operators, layers, operation bodies, and tasks keep their extension + code loaded. + +## Current Implementation Evidence + +The proposal derives its constraints from these current implementations: + +- The [core split RFC](../../../core/core/src/docs/rfcs/6828_core.md) prepares + service and layer crates for a future extension ecosystem but leaves dynamic + loading unresolved. +- The current [operator registry](../../../core/core/src/types/operator/registry.rs) + stores plain factories, while + [`OperatorUri`](../../../core/core/src/types/operator/uri.rs) defines the + initial URI and explicit-option merge. +- The native [`Layer` trait](../../../core/core/src/raw/layer.rs) composes both + service and operation-context planes. +- [Timeout](../../../core/layers/timeout/src/lib.rs), + [Foyer](../../../core/layers/foyer/src/lib.rs), and + [Throttle](../../../core/layers/throttle/src/lib.rs) exercise context + replacement, asynchronous state, and shared state respectively. +- The [S3 configurator](../../../core/services/s3/src/config.rs) and + [WebDAV configurator](../../../core/services/webdav/src/config.rs) demonstrate + why each service must own URI interpretation. +- The [HDFS service](../../../core/services/hdfs/README.md) documents its + libhdfs/JVM environment, which must remain outside the base binding. +- The [Python](../../python/Cargo.toml), [Ruby](../../ruby/Cargo.toml), and + [Node.js](../../nodejs/Cargo.toml) manifests show the current monolithic + feature and native-library layouts. +- The [security threat model](../../../SECURITY-THREAT-MODEL.md) treats binding + ownership, lifetime, and safe FFI behavior as part of OpenDAL's contract. + +## Candidate Architecture + +```text +Python package Ruby gem npm package + | | | + v v v +Python adapter Ruby adapter Node-API adapter + \ | / + +-----------------+--------------------+ + | + v + OpenDAL extension runtime + registry, Tokio, core types, loader, + errors, handles, library leases + / \ + v v + service extension layer extension + S3 / WebDAV / HDFS Timeout / Foyer / Throttle +``` + +The diagram shows a logical architecture. The first implementation may package +a separate runtime artifact for each language ecosystem. Sharing the SDK, +descriptor, rules, and conformance suite is required; sharing one physical +runtime library across Python, Ruby, and Node.js installations is not. + +### Binding adapter + +Each binding adapter translates language values, async behavior, exceptions, +and garbage collection into runtime-owned handles. It does not implement +service configuration or native layer composition. + +The adapter preserves each language's normal interface: + +- Python keeps `Operator`, `AsyncOperator`, `await`, type stubs, and Python + package discovery. +- Ruby keeps blocking `Operator` operations, `require`, keyword arguments, and + Ruby exceptions. +- Node.js keeps promises, synchronous variants where supported, JavaScript + streams, ESM/CommonJS exports, and Node-API. + +The project should share the native extension contract, not force one public +language API across all three bindings. The manifest, bootstrap descriptor, +configuration value grammar, factory semantics, error categories, and +conformance suite are common. Constructor names, typing, duration syntax, +blocking behavior, and package discovery remain binding-specific. + +### Extension runtime + +The runtime module owns process-scoped native resources. Python and Ruby +initially use one `NativeRuntime` per loaded runtime module and process. The +Node.js runtime calls that owner `ProcessRuntime` and places one +`EnvironmentAdapter` per `napi_env` over it. + +The runtime module owns: + +- The OpenDAL core and Tokio runtime used by extension objects. +- Service and layer registries. +- Scheme dispatch while preserving the original construction request for + package-owned URI parsing and configurators. +- Operator, layer, and operation-body handles. +- Panic containment and language-neutral error details. +- Native library activation and process-lifetime pinning. + +The current `OperatorRegistry` is not sufficient for this role. It stores +plain function pointers and replaces an existing scheme during registration. +The extension registry must record package ownership and perform atomic, +conflict-detecting registration. + +### Language package + +Each installable package contains: + +- Language code or type declarations. +- A text manifest with the package ID, provided schemes or layers, native + artifact path, unique entry symbol, and required runtime build ID. +- A target-specific native library, or source needed to build one. +- Package-specific documentation and conformance tests. + +Registering the text manifest must not load the native library. The runtime +activates the library when a caller first constructs a declared service or +layer. This isolates HDFS and similar native dependencies. + +### Native extension + +A native extension contains one or more service or layer factories. The +package exports one generated, package-unique bootstrap symbol and hides other +package-local symbols where the platform permits it. + +The bootstrap uses a small C-layout descriptor so the runtime can inspect its +size, revision, manifest identity, target, and runtime build ID safely. Only +after all checks pass may the runtime enter the exact-build internal interface. +That internal interface is not a stable Rust ABI. + +Node.js packages are an exception to the unique-initializer rule when the +native artifact is itself a Node-API addon: Node defines the addon initializer. +The Node adapter must use a language-appropriate initializer and validate the +same package identity in the returned descriptor. + +## Construction Requests + +A service factory accepts one of two request forms: + +```text +UriRequest { + raw_uri, + explicit_string_options, +} + +ConfigRequest { + package_schema_revision, + structured_values, +} +``` + +`structured_values` uses the versioned, language-neutral `ConfigValue` grammar +defined by [the compatibility contract](compatibility.md#configuration-value-contract). +Each binding validates and converts its public values before invoking a native +factory; a native package never receives Python, Ruby, or JavaScript objects. + +The URI request preserves the original URI and explicit options. The package +constructs its local `OperatorUri` and calls its own `Configurator::from_uri`. +The runtime must not replace service-specific behavior with a universal +precedence rule. + +A layer factory accepts structured configuration and returns an asynchronous +result: + +```text +create_layer(layer_id, structured_values) -> future +``` + +`LayerHandle` retains state and can be applied to more than one operator. +Applying a layer returns a new operator. The runtime never sorts layers: later +applications are outer layers, matching OpenDAL core behavior. + +## Registration Rules + +- Scheme and layer IDs use lowercase canonical strings. +- One manifest can claim several names and aliases atomically. +- Re-registering the identical package and manifest is idempotent. +- A different owner claiming an existing name rejects the entire manifest. +- Registry locks are not held while loading code or running a factory. +- Capabilities are read from a constructed operator, not declared statically in + registration metadata. +- A failed native activation affects only the requested package. +- Extensions remain loaded for the rest of the process in the first version. + +## Package Families + +Each ecosystem needs a runtime distribution, independent extension packages, +and a compatibility aggregator: + +```text +runtime distribution minimal binding and native runtime +service/layer packages exact dependency on that runtime release +compatibility aggregator runtime plus the services/layers shipped today +``` + +The aggregator preserves existing installation commands during migration. A +minimal installation uses the runtime distribution plus selected extension +packages. Independently installable does not imply independently +ABI-versioned: every exact-runtime native package must be rebuilt for a new +runtime build ID. + +The 1,000-manifest requirement measures registry behavior, not a commitment to +publish 1,000 official package families. Before a split release, each ecosystem +needs reserved package names, trusted publishing, coordinated release tooling, +and rollback rules. OpenDAL can publish only supported high-value extensions +while leaving the same SDK available to third parties. + +## Security Model + +Native extensions are trusted in-process code. Compatibility validation +prevents accidental mismatches; it does not sandbox, authenticate, or constrain +a malicious package. + +The host still owns these in-scope properties: + +- No panic or language exception crosses any FFI boundary. +- Safe binding operations do not cause use-after-free or data races. +- Official packages redact credentials from errors and diagnostics. +- Registry metadata and build diagnostics do not contain secrets. +- An extension callback cannot run after its library is unloaded. + +The loader must use an explicit package-provided artifact path. It must not scan +arbitrary library search paths and execute every matching file. + +## Delivery Gates + +The design is not ready for an RFC decision until prototypes demonstrate: + +1. A minimal runtime plus separately packaged FS and Retry tracers. Memory + remains in the runtime because OpenDAL core always provides it. +2. S3 and WebDAV URI behavior without host-owned configuration schemas. +3. Lazy HDFS failure without affecting S3, WebDAV, or `hdfs-native`. +4. Complete Timeout context behavior and shared Throttle state. +5. Asynchronous, stateful Foyer construction and lifetime handling. +6. Atomic registration and lazy lookup with 1,000 synthetic packages. +7. Target packaging on supported Linux, macOS, and Windows variants. +8. Python, Ruby, and Node.js adapters passing the same native conformance suite. + +If those gates pass, the next deliverable is a `0000_*.md` RFC. These pre-RFC +documents remain supporting analysis rather than evidence that the candidate +has already been accepted. diff --git a/bindings/docs/dynamic-extensions/alternatives.md b/bindings/docs/dynamic-extensions/alternatives.md new file mode 100644 index 000000000000..0e493e26cd12 --- /dev/null +++ b/bindings/docs/dynamic-extensions/alternatives.md @@ -0,0 +1,333 @@ + + +# Dynamic Extension Design Alternatives + +Status: pre-RFC design comparison. No alternative has been accepted or +implemented. + +This document compares four ways to deliver OpenDAL services and layers to +language bindings. The designs place the extension seam at different levels, +which changes their native fidelity, compatibility, and implementation cost. + +## Decision Guide + +```text +Must packages compose after installation? + no -> Design A: feature-selected monolithic binding + yes -> Must arbitrary native layers preserve core semantics? + no -> Design B: language-owned operator adapters + yes -> Must an old binary survive runtime/core upgrades? + no -> Design C: exact-build shared runtime + yes -> Design D: stable C operation interface +``` + +OpenDAL should prototype Design C. Design A remains useful for custom builds. +Design B does not meet the native-layer requirement. Design D should be chosen +only if cross-version third-party binary compatibility becomes a strategic +requirement that justifies stabilizing a second operation model. + +## Summary + +| Property | A. Static build | B. Language adapters | C. Exact runtime | D. Stable C interface | +| ------------------------------ | ----------------- | -------------------- | ---------------------- | -------------------------- | +| Independently install services | No | Yes | Yes | Yes | +| Independently install layers | No | Language decorators | Yes | Yes | +| Full native layer semantics | Yes | No | Yes | Only represented semantics | +| One core/runtime graph | Yes | Usually no | Yes | Usually no | +| Third-party source extensions | Rebuild host | Yes | Yes, exact build | Yes | +| Cross-version native binaries | Not applicable | Language-dependent | No | Yes, within ABI rules | +| Cross-language native package | No | No | Possible, not promised | Yes, by design | +| Native dependency isolation | No | Yes | Yes | Yes | +| Implementation cost | Lowest | Moderate | High | Highest | +| Primary risk | Artifact variants | Semantic loss | Packaging/linking | ABI breadth and safety | + +## Design A: Feature-Selected Monolithic Binding + +### Seam + +Cargo features select services and layers when the language extension is +compiled. The resulting wheel, gem, or npm native package contains one OpenDAL +core and all selected implementations. + +```text +language caller -> one native binding -> compiled services and layers +``` + +### Strengths + +- Preserves all native behavior without a new runtime interface. +- Uses ordinary Rust ownership and one Tokio/core graph. +- Has the smallest implementation and verification cost. +- Works well for downstream users who build one controlled deployment image. + +### Limitations + +- A service cannot be added after the binding is built. +- Two feature variants normally provide the same import/module name and cannot + be installed together. +- One uncommon native dependency can constrain the whole artifact. A build + containing libhdfs-backed HDFS may fail to load where Java/Hadoop libraries + are absent. +- A large published feature set increases build time, artifact size, supply + chain surface, and platform exclusions. + +### Appropriate Use + +Keep this path for custom source builds and hermetic distributions. It does not +meet the independently installable package requirement. + +## Design B: Language-Owned Operator Adapters + +### Seam + +Each service package owns an operator and exposes a Python, Ruby, or JavaScript +operation protocol. The base binding delegates every operation through that +language protocol. Layers decorate language objects. + +```text +language Operator facade -> language operation protocol -> package-owned backend + ^ + | + language decorator +``` + +### Strengths + +- Package loading uses normal language mechanisms. +- Pure-language third-party services need no Rust ABI. +- Each native package can own its core version and runtime because Rust values + do not cross the seam. +- HDFS dependencies remain isolated in the HDFS package. + +### Limitations + +- Every service package repeats or generates the complete operation adapter. +- Language calls appear in streaming and operation-body paths unless the + adapter adds another native batching interface. +- A native OpenDAL layer cannot wrap an operator owned by another extension. +- Reproducing native layers eventually requires readers, writers, listers, + deleters, copiers, cancellation, HTTP context, and executor context. At that + point this design has recreated a language-specific version of Design D. + +### Constraint Failures + +Timeout is decisive. `TimeoutLayer` wraps service calls and replaces the +executor in `OperationContext`. A method decorator cannot reproduce the +executor behavior used by concurrent block operations. + +Foyer is also stateful and intercepts operation bodies. A language decorator +can implement a different cache, but it cannot claim native `FoyerLayer` +equivalence without a much larger protocol. + +### Appropriate Use + +Use this design when service extensibility matters but native layer parity does +not. It does not meet this proposal's layer requirement. + +## Design C: Exact-Build Shared Native Runtime + +### Seam + +The binding adapter and all native packages use one shared native runtime +module. That module owns OpenDAL/Tokio types and registries. Extensions register +erased service or layer factories compiled for exactly that runtime build. +Node.js can place an environment adapter over those process-scoped resources; +it does not create a second OpenDAL graph for each Worker. + +```text +language adapter -> extension runtime <- service/layer native libraries + | + v + runtime-owned Operator +``` + +The loader first reads a stable, C-layout bootstrap descriptor. It validates an +exact runtime build ID before entering any build-specific internal interface. + +### Strengths + +- Preserves the native `Operator`, `Layer`, and `OperationContext` model. +- Keeps one OpenDAL core, Tokio runtime, and shared HTTP/context graph. +- Makes service and layer packages small and keeps configuration local. +- Supports stateful layer handles and asynchronous layer construction. +- Isolates native dependencies through lazy activation. +- Shares the extension SDK and conformance suite across language bindings. + +### Limitations + +- Rust ABI stability is not promised. Every native package rebuilds for each + runtime build ID. +- Cross-platform shared-library discovery, repair, rpath/install-name behavior, + Windows DLL lookup, and symbol visibility require substantial engineering. +- Package managers must prevent or clearly reject mixed runtime release trains. +- One physical extension artifact shared across language ecosystems requires proof; + the initial design only requires shared source and contracts. +- Native packages remain trusted code and cannot be unloaded safely in the + initial design. + +### Appropriate Use + +This is the recommended balance when install-time composition and complete +native layers matter, but old extension binaries do not need to survive runtime +upgrades. + +## Design D: Stable C Operation Interface + +### Seam + +The host and each native extension exchange only C-layout function tables, opaque +reference-counted handles, fixed-width scalars, buffers, errors, and versioned +wire values. Rust values never cross the seam. + +```text +language adapter -> host operation graph -> stable C ABI -> extension operation graph +``` + +To support arbitrary services and layers, the interface must represent: + +- Service construction, capabilities, and every operation. +- Readers, writers, listers, deleters, copiers, and streams. +- Futures, polling or completion, waking, and cancellation. +- Buffer ownership and creator-side destruction. +- Errors and extensible operation options. +- HTTP and executor resources in `OperationContext`. +- Separate service and context hooks for layers. + +### Strengths + +- Extensions can use different OpenDAL core and Rust compiler versions. +- A correctly versioned extension binary can survive host upgrades. +- Native extensions can be shared by Python, Ruby, and Node.js on one target. +- Other C-ABI languages can implement extensions. + +### Limitations + +- The C interface becomes a second stabilized representation of OpenDAL's raw + operation model. +- Async cancellation, stream ownership, context composition, and wire evolution + create a large unsafe verification surface. +- Old extensions may be binary compatible while lacking new operation semantics. + They must reject required fields they do not understand rather than silently + dropping them. +- Each extension may embed its own OpenDAL/Tokio graph, increasing size and + making cross-extension layers more expensive. + +### ABI Evolution + +If OpenDAL selects this design, it must govern the operation ABI independently +from every language binding: + +- Every function table starts with a structure size and ABI version. +- A minor ABI revision can append optional methods but cannot reorder fields or + change existing ownership or semantics. +- A breaking layout, ownership rule, or required semantic change increments the + ABI major. +- Extensible wire values distinguish required and optional fields. An old + extension rejects an unknown required field instead of silently losing it. +- Every output has a creator-provided destructor. +- Task polling and mutable operation bodies define concurrency and cancellation + explicitly. +- Loader negotiation checks ABI major, wire major, target identity, and required + interface IDs before it exchanges handles. + +Stable ABI compatibility does not mean that an old extension implements every +new operation. It means the host can load the extension safely, negotiate its +capabilities, and receive an explicit unsupported result. + +### Appropriate Use + +Choose this design only when the project commits to long-term binary extension +governance. A service-only C interface is not enough for the layer requirement. + +## Constraint Comparison + +### S3 and WebDAV + +All designs must keep URI interpretation inside the service implementation. +The shared host passes a raw URI and explicit options instead of maintaining a +central configuration schema. + +This matters because generic precedence is insufficient: + +- `OperatorUri` applies query options first and explicit options second. +- S3 then derives bucket and root from the URI name and path. +- WebDAV derives `https://authority` as its endpoint, overwriting an endpoint + option supplied earlier. + +Designs B, C, and D can preserve these rules if the package owns construction. +A host-owned universal schema is rejected in every design. + +### HDFS + +Design A cannot guarantee that the base artifact loads without HDFS native +dependencies when HDFS is compiled into it. Designs B, C, and D can register a +lightweight manifest and activate HDFS only at construction. `hdfs` and +`hdfs-native` remain independent packages and schemes. + +### Foyer + +Foyer requires asynchronous cache creation and a reusable stateful handle. +Design C holds the native layer in the shared runtime. Design D represents it +through a reference-counted layer handle. Design B can provide a language-level +cache but cannot claim arbitrary native layer composition. + +Foyer's current key uses path and optional version, not service identity. No +design should advertise one cache handle as isolated across unrelated storages +until the package adds namespacing or restricts the handle to one logical +storage. + +### Timeout and Throttle + +Designs A, C, and D can preserve both Timeout service wrapping and executor +wrapping. Design B cannot do so through method delegation alone. + +One Throttle layer owns shared limiter state. Designs C and D preserve the +identity of the layer handle so applying it to two operators shares a quota. +The extension adapter must validate positive bandwidth and burst values before +calling a constructor that currently asserts them. + +Timeout must be inside Retry. Applying Timeout and then Retry gives each retry +attempt a deadline. Applying Retry and then an outer Timeout is cancellation +unsafe because Timeout can drop Retry before it restores operation-body state. +An extension design must preserve the safe order and must not advertise an +outer whole-sequence Timeout as equivalent. + +## Why Node.js Does Not Change the Decision + +[Node-API](https://nodejs.org/api/n-api.html) is ABI-stable across supported +Node.js versions, so it is a strong seam between JavaScript and the base addon. +It does not stabilize OpenDAL's internal Rust types or external libraries used +by the addon. Therefore Node.js benefits from Design C for the same reason as +Python and Ruby. + +Node.js already publishes target-specific native npm packages, which provides a +useful packaging pattern for extension artifacts. It still needs an exact +runtime build check, lazy activation, per-Worker adapter state, and a clear rule +against transferring JavaScript wrappers between Node environments. + +## Recommendation + +Prototype Design C with one language-neutral extension SDK and three binding +adapters. Preserve Design A as a supported custom-build path. Do not combine C +with B: a language-owned operator plus an exact-runtime capsule creates two +competing composition models and inherits both sets of lifetime rules. + +Reconsider Design D only if users require already-built third-party extensions to +survive OpenDAL runtime upgrades. That requirement should be explicit because +it changes the project from coordinating a release train to governing a stable +operation ABI. diff --git a/bindings/docs/dynamic-extensions/compatibility.md b/bindings/docs/dynamic-extensions/compatibility.md new file mode 100644 index 000000000000..1cfd8618ade9 --- /dev/null +++ b/bindings/docs/dynamic-extensions/compatibility.md @@ -0,0 +1,536 @@ + + +# Extension Compatibility and ABI + +Status: pre-RFC contract for the exact-build runtime candidate. OpenDAL does not +provide or guarantee this ABI today. + +This document distinguishes public language compatibility, package discovery, +the safe loader bootstrap, and the build-specific native interface. A version +match in one category does not imply a match in another category. + +The key constraint is deliberate: + +> The candidate design provides a stable language interface and a stable +> loader handshake, but it does not provide a stable Rust extension ABI. + +Native extensions must match an exact runtime build ID. An extension rebuild is +required whenever that build ID changes. + +## Compatibility Identities + +| Identity | Purpose | Stability rule | +| ------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------- | +| Binding interface version | Public Python, Ruby, or Node.js behavior | Governed independently by each binding's compatibility policy | +| Manifest revision | Text discovery and registration fields | Major for breaking syntax/meaning; minor additions must be ignorable | +| Extension SDK revision | Source interface used to build native extensions | Semantic versioning can describe source changes, but not native compatibility | +| Bootstrap ABI version | C-layout descriptor read before native activation | Stable major; minor revisions append bounded optional fields | +| Runtime build ID | Exact internal native interface and shared Rust types | Equality required; no compatibility range | +| Target identity | OS, architecture, ABI, libc/deployment floor, and target features | Must be compatible with the running process and runtime artifact | + +Equal OpenDAL versions, binding versions, Rust compiler versions, or SDK +revisions are individually insufficient to establish exact-runtime binary +compatibility. + +## Public Language Interface + +The public binding interface remains independent in each ecosystem: + +- Python versions `Operator`, `AsyncOperator`, exceptions, typing, and package + discovery according to the Python binding policy. +- Ruby versions `OpenDal::Operator`, layers, errors, and blocking behavior + according to the Ruby binding policy. +- Node.js versions JavaScript classes, promises, synchronous variants, streams, + and TypeScript declarations according to the Node.js binding policy. + +A public binding release may preserve all language-level behavior while +changing its runtime build ID and requiring every native extension to rebuild. +Conversely, an extension package can change a typed configuration helper +without changing the runtime ABI. + +Python `abi3` describes compatibility between a Python extension and CPython +versions. A [Python capsule](https://docs.python.org/3/c-api/capsule.html) can +carry an opaque bootstrap pointer through the stable CPython ABI. Neither fact +stabilizes Rust values behind that pointer. + +[Node-API](https://nodejs.org/api/n-api.html) describes compatibility between +the base addon and Node.js versions. Its own documentation explicitly excludes +external libraries from that guarantee. It does not make two napi-rs +`External` values from different addons interchangeable. + +The Ruby adapter remains subject to the Ruby and Magnus compatibility policy. +A language-neutral service/layer extension should not link Ruby and therefore +does not acquire a Ruby ABI dependency of its own. + +## Configuration Value Contract + +Factories receive language-neutral configuration values. Version 1 has this +closed grammar: + +```text +ConfigValueV1 = + Null + | Bool(bool) + | I64(i64) + | U64(u64) + | F64(finite IEEE-754 binary64) + | Utf8(string) + | Bytes(byte sequence) + | List(sequence) + | Map(map) + | DurationNs(u64) +``` + +`DurationNs` is the only duration representation at the shared factory seam. +It covers zero through `u64::MAX` nanoseconds. Binding-specific interfaces can +accept seconds, milliseconds, or native duration objects, but their adapters +must define nanosecond rounding and reject negative, non-finite, or overflowing +values before producing `DurationNs`. Numeric conversion never silently +truncates, saturates, or wraps. + +The grammar has its own major version. Adding a value variant, changing a +numeric interpretation, or changing container semantics creates a new major and +a new exact runtime build ID. A new runtime can implement more than one grammar +major explicitly, but it never reinterprets an old tag. + +The decoder enforces a maximum nesting depth of 32, a maximum UTF-8 map key of +4 KiB, a maximum individual string or byte value of 16 MiB, a maximum of 65,536 +entries in one list or map, and a maximum encoded request size of 64 MiB. It +rejects invalid UTF-8, duplicate map keys, non-finite floats, unknown value tags, +and values outside the declared numeric ranges before calling package code. + +A package versions its configuration schema independently from this value +grammar. Schema behavior follows these rules: + +- A missing field selects its declared default or produces a missing-field + error. +- `Null` is valid only when the field is explicitly nullable. Optional and + nullable are separate properties. +- A package rejects every field absent from its schema. Version 1 has no + implicit or client-declared "ignorable unknown" convention. +- A package rejects an unsupported schema revision. It can declare an explicit + set or range of revisions only when it implements each revision's defaults + and field meanings. +- A schema marks credential and token fields as secret. The runtime never logs + raw configuration values, and package errors identify a secret field without + rendering its value. + +`UriRequest` remains separate: it contains one UTF-8 URI and a map of UTF-8 +option names to UTF-8 option values, matching current iterator construction. +The runtime applies the same size limits and never includes raw option values in +loader diagnostics. Service code remains responsible for URI semantics and +service-specific validation. + +## Manifest Compatibility + +Language packages register a text manifest without activating native code. A +manifest contains at least: + +```text +manifest_revision +package_id +package_version +provided_services[] +provided_layers[] +native_artifact_path +native_entry_kind +native_entry_symbol +runtime_build_id +target_identity +manifest_digest +``` + +The registry must validate sizes, encoding, IDs, duplicate names, and paths +before storing the manifest. It compares the declared runtime build ID and +target with the running artifact and rejects a mismatch without loading native +code. It must not store credentials, URI options, or service configuration in +registration metadata. + +Manifest evolution follows these rules: + +- A new major revision may add a required field or change field meaning. +- A new minor revision may add optional fields only. Unknown fields in a newer + minor are optional by definition and an older reader ignores them. +- Readers reject an unknown major revision. +- Writers do not omit a field required by their declared major revision. +- A package's embedded native descriptor must repeat the package identity, + runtime build ID, target identity, and manifest digest. Activation fails if + the text and native declarations differ. + +## Bootstrap ABI + +The bootstrap exists only to inspect and reject an extension safely before +entering the exact-build interface. It must not expose `Operator`, `Layer`, +trait objects, Rust strings, Rust enums, futures, Tokio handles, or allocator +ownership. + +An illustrative descriptor is: + +```c +typedef struct { + const unsigned char *data; + size_t len; +} opendal_bytes_v1; + +typedef struct { + uint64_t magic; + uint32_t struct_size; + uint16_t bootstrap_major; + uint16_t bootstrap_minor; + opendal_bytes_v1 package_id; + opendal_bytes_v1 manifest_digest; + opendal_bytes_v1 runtime_build_id; + opendal_bytes_v1 target_identity; + void (*exact_build_entry)(void); +} opendal_extension_header_v1; +``` + +The actual RFC must specify maximum lengths, pointer validity, descriptor +lifetime, encoding, alignment, and failure behavior. Descriptor memory should +be immutable and valid while the native library remains loaded. + +The bootstrap ABI follows these rules: + +- Every exported function uses an explicit C calling convention. +- No panic or foreign exception crosses the call. +- Every structure begins with a size and version or is referenced by a + versioned descriptor. +- Minor versions append fields; they never reorder or reinterpret existing + fields. +- The loader checks `magic`, minimum size, major version, bounded byte slices, + target identity, manifest digest, and runtime build ID in that order. +- The loader does not call `exact_build_entry` after any failed check. +- The bootstrap reports incompatibility; it does not attempt ABI adaptation. + +This stable bootstrap is intentionally too small to construct an operator. A +stable construction interface would be Design D from +[the alternatives](alternatives.md), not an incremental relaxation of the +exact-build rule. + +## Runtime Build ID + +The coordinated runtime build issues an opaque, collision-resistant build ID. +Extension authors must not type or copy that identity into source code. The +published SDK build tool validates the extension build inputs against the +runtime build manifest and generates both the embedded descriptor and the text +manifest. It fails the build if a required input cannot be verified. + +The build manifest behind the ID must cover all ABI-relevant inputs, +including: + +- Extension runtime and SDK layout revisions. +- Exact OpenDAL core source identity. +- Exact versions and features of dependencies whose types or globals cross the + seam. +- `rustc -vV`, target specification, target features, and ABI-relevant codegen + options. +- Panic strategy, allocator/ownership policy, and symbol/linkage model. +- The supported factory and handle layouts. + +Package-local dependencies such as S3 signing, XML parsing, `hdrs`, Foyer, or a +rate limiter do not need identical versions when their types and globals stay +inside the package. If one of their types crosses the seam, it becomes an +ABI-visible dependency and must enter the runtime build identity. + +The supported SDK is a coordinated build tool, not only a Rust crate. It pins +the exact SDK dependencies and lock state, checks the compiler and target, +normalizes the permitted Cargo profile and Rust flags, verifies the panic and +linkage policy, and records the actual inputs in a machine-readable build +report. Release CI compares that report with the generated descriptor before it +publishes an artifact. + +Build-ID equality is a necessary accidental-mismatch check, not attestation and +not proof that arbitrary native code is compatible or well behaved. Official +packages use the coordinated release pipeline. A supported third-party source +package must build through the published SDK toolchain and pass the same +conformance suite. An arbitrary binary that merely claims or forges an issued +ID is trusted, unsupported in-process code; the loader cannot make it safe. + +The project must never replace a published runtime artifact with different bits +or a different build ID under the same package version. A rebuild receives a +new immutable runtime release. + +## Change Impact + +| Change | Binding interface version | Runtime build ID | Extension action | +| ------------------------------------------------------------- | ------------------------------------------ | ---------------- | -------------------------------------------------- | +| Install another compatible service/layer | Unchanged | Unchanged | Register the new manifest | +| Add an optional service config field | Service package policy | Unchanged | Republish only that package if native code changes | +| Change a public language method incompatibly | Binding policy requires a breaking release | Maybe | Rebuild only if native inputs change | +| Add an optional manifest field | Manifest minor | Unchanged | Old readers ignore it | +| Append a bootstrap descriptor field | Bootstrap minor | Unchanged | Old loaders use the recorded structure size | +| Change an exact SDK handle or factory layout | Unchanged or additive language release | New ID | Rebuild every native extension | +| Change ABI-visible OpenDAL core code or features | Usually unchanged at language level | New ID | Rebuild every native extension | +| Change `rustc`, target ABI, panic strategy, or linkage inputs | Unchanged | New ID | Rebuild every native extension | +| Change only a package-private dependency | Unchanged | Unchanged | Rebuild only that package | + +This table describes compatibility impact, not release numbering. Each binding +and package still follows its own published version policy. + +## Exact-Build Internal Interface + +After the bootstrap succeeds, the runtime may use a build-specific factory +interface generated by the extension SDK. That interface can exchange +runtime-owned operator and layer handles or exact-build Rust adapters. + +The following constraints apply: + +- The interface is compatible only with one runtime build ID. +- The [Rust ABI has no stability guarantees](https://doc.rust-lang.org/reference/items/external-blocks.html). +- Extensions rebuild for every new runtime build ID, including patch releases when + the ABI-relevant build changes. +- Equal layouts observed in two builds do not create a supported compatibility + range. +- The loader rejects a mismatch before registering factory pointers. +- Package-local code catches panics before returning across the seam. +- Ownership remains on the creating side unless an exact SDK handle explicitly + transfers it. + +The first implementation should prefer opaque runtime handles and generated +adapters even inside the exact-build interface. This reduces the number of Rust +types that can accidentally become ABI-visible and makes a future stable ABI +easier to evaluate. + +## Target Identity + +A runtime build ID does not make an artifact portable across targets. The +loader must also match: + +- Operating system and architecture. +- Pointer width, endianness, and calling convention. +- Linux libc family and minimum version where applicable. +- macOS deployment target and architecture. +- Windows toolchain/runtime family. +- Required CPU target features. +- Language adapter variant when the runtime is not physically shared. + +A runtime release may issue different build IDs for Python, Ruby, and Node.js +even when all three use the same extension SDK source. Cross-language reuse of +one physical native artifact is supported only after artifact-level tests prove +that those adapters load the same runtime identity on that target. + +## Loader State Machine + +```text +unregistered + | + | register validated text manifest + v +registered + | + | first construction + v +loading -> bootstrap validated -> factories installed -> active + | | | + +---- error -----+----------------------+--> failed +``` + +The loader must provide these properties: + +1. Registration of all names in one manifest is atomic. +2. An identical registration is idempotent. +3. A different owner for an existing name rejects the full registration. +4. Registry locks are released before filesystem access, `dlopen`, addon + loading, or package code. +5. Only the requested package activates. +6. One failed package does not poison unrelated registrations. +7. Concurrent first construction activates a package once and shares the + result or failure deterministically. +8. The runtime retains a library lease before storing any callback. +9. The initial implementation never unloads an activated native extension. + +The current core registry silently replaces an existing scheme. The extension +registry must not inherit that behavior. + +## Package Manager Constraints + +Package metadata provides an early compatibility diagnostic. The native loader +remains authoritative because package managers do not understand the embedded +runtime build ID. + +| Ecosystem | Proposed constraint | Additional requirement | +| --------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| Python | Extension distribution requires an exact `opendal-runtime` release | Embedded build ID check; wheel target must match | +| Ruby | Extension gem requires an exact `opendal-runtime` release | Embedded build ID check; source/native gem policy remains explicit | +| Node.js | Extension uses an exact runtime peer dependency and target-specific optional packages | Embedded build ID check; reject a nested wrong `ProcessRuntime` | + +Extension package versions may evolve independently from their target runtime. +Each published version still declares exactly which runtime release it targets. +A service fix that does not change the runtime can publish a new service package +against the same runtime build ID. + +The compatibility aggregator named `opendal` installs the runtime and the +service/layer set provided by the current monolithic release. Minimal users +install the runtime distribution and selected extensions directly. Release +publication order is runtime, extensions, then aggregator. + +Dependency installation alone does not register an extension. Each aggregator +must guarantee registration using its binding's defined mechanism: mandatory +named entry-point resolution in Python, a generated registration-stub index in +Ruby, and explicit stub imports during Node.js module initialization. The +aggregator must register metadata only and preserve construction-time native +activation. + +## Service Compatibility Rules + +- A service factory receives the raw URI and explicit string options. +- The service package owns `Configurator::from_uri`, aliases, validation, + credentials, and redaction. +- Query options are merged before explicit options by `OperatorUri`, but the + service configurator may subsequently derive or replace fields from URI + authority and path. +- Capabilities belong to the constructed operator. Registration metadata does + not promise static capabilities. +- A package retains every native resource used by active operations. +- Construction errors identify the package and operation without copying + secrets into loader diagnostics. + +S3 must continue deriving bucket and root from its URI. WebDAV must continue +deriving an HTTPS endpoint from authority, including its current behavior of +overwriting an endpoint option. HDFS activation must be lazy and isolated from +`hdfs-native`. + +## Layer Compatibility Rules + +- A layer factory may complete asynchronously. +- A layer handle retains native state and its extension library lease. +- A layer descriptor carries its canonical ID and declared composition safety + constraints. An operator handle retains the ordered IDs applied through the + supported binding interface. +- Applying a layer returns a new operator and does not mutate the source + operator. +- Later applications are outer layers. The runtime does not sort or deduplicate + layers. +- Both `apply_service` and `apply_context` behavior must be preserved. +- Reusing one handle preserves that layer's sharing identity. + +Timeout and Retry have one cancellation-safe order. Applying Timeout first and +Retry second places Retry outside Timeout, so each retry attempt has its own +timeout. Applying Retry first and Timeout second places Timeout outside Retry; +the timeout can drop Retry's future before Retry restores operation-body state. +Official bindings must preserve caller order and reject this known unsafe outer +Timeout composition when the operator's layer metadata makes it visible. They +must document it as unsupported when an opaque third-party layer prevents that +check. The runtime must never silently reorder the layers. + +Foyer live cache state and Throttle token history are not automatically +serializable. A package can define a declarative reconstruction policy, but it +must distinguish a reconstructed resource from the original live state. + +## Lifetime and Concurrency + +- Operator and immutable layer handles may be shared according to the native + OpenDAL contract. +- Mutable readers, writers, listers, deleters, copiers, and task handles must + document and enforce their allowed concurrency. +- A language wrapper can be collected while a derived operator remains active; + the runtime handle must keep native state alive. +- Cancellation must drop or abort the represented operation according to its + documented semantics. It must not detach work merely to simplify FFI. +- Destruction runs through the creator's function or exact SDK adapter. +- No callback, destructor, or wake operation may target an unloaded library. +- Reuse after `fork` is unsupported for Tokio, JVM, connection-pool, and cache + state unless a package explicitly documents and tests it. + +Node.js adds one constraint: JavaScript and Node-API values belong to one +`napi_env`. A loaded runtime native module owns a process-scoped +`ProcessRuntime`, including registries, native handle identity, activation +state, and library leases. Each environment owns a separate +`EnvironmentAdapter` and cleanup hooks. The adapter must not move +environment-bound values between the main thread, Workers, or Electron +contexts. + +## Error Contract + +The shared runtime must expose stable error categories for: + +- Manifest invalid or conflicting registration. +- Package not installed or not registered. +- Native artifact missing or unsupported on this target. +- Bootstrap symbol or descriptor invalid. +- Runtime build ID, target, or manifest mismatch. +- Native dependency load failure. +- Service/layer configuration invalid. +- Asynchronous layer initialization failure or cancellation. +- Package panic contained at the extension seam. +- Normal OpenDAL operation errors. + +Bindings map those categories into language-native exception classes while +retaining package ID, scheme/layer ID, and construction operation. Diagnostics +must not include credentials or an unredacted configuration map. + +## Security Constraints + +Native extensions are trusted in-process code. Neither the bootstrap nor an +exact build ID provides a sandbox, signature verification, provenance, process +isolation, or protection from a malicious extension. + +Compatibility validation still protects the supported binding contract from +accidental mismatches. The loader and official SDK must: + +- Validate all untrusted lengths and identifiers before use. +- Catch panics at every exported native entry and runtime-invoked callback. +- Keep package symbols local where supported. +- Load only a path supplied by an installed, explicitly selected package. +- Pin libraries while any callback, vtable, task, or object may reference them. +- Keep credentials out of manifests, fingerprints, and conflict errors. + +Package signing and registry provenance remain package-release concerns. They +do not change the in-process trust model. + +Official runtime and extension artifacts must use unwind-capable panic handling +at extension entries. An artifact built with `panic=abort` cannot contain a +panic at the seam and still satisfy containment; the SDK and release pipeline +therefore reject that panic strategy for this interface. The build ID records +the strategy so a mismatched artifact also fails compatibility validation. + +## Unsupported Guarantees + +The exact-build design does not guarantee: + +- Loading an extension built for a different runtime build ID. +- Compatibility based only on equal OpenDAL or binding semantic versions. +- Native library unloading. +- Safe use of arbitrary extension code or secret redaction by third parties. +- Serialization of live caches, limiters, runtimes, JVMs, or connection pools. +- One physical extension binary across Python, Ruby, and Node.js. +- Static service capabilities before configuration. +- Runtime installation of a missing package. Discovery only observes packages + already installed by the application's package manager. +- Backend correctness, authorization, durability, or resource limits beyond + the existing OpenDAL threat model. + +## Required Conformance + +Before publishing a native extension interface, CI must verify: + +- 1,000 manifest registrations without eager native activation. +- Atomic multi-name conflicts and idempotent duplicate registration. +- Mismatch rejection before storing callbacks. +- Concurrent activation and process-lifetime library pinning. +- S3 and WebDAV URI/configuration semantics and secret-free errors. +- HDFS load isolation from the base, S3, WebDAV, and `hdfs-native`. +- Foyer asynchronous construction, cancellation, invalidation, and handle + lifetime. +- Timeout service and executor behavior through a separately packaged layer. +- Timeout-inside-Retry acceptance and unsafe outer-Timeout rejection. +- Throttle shared limiter identity and argument validation. +- Language garbage collection, task cancellation, and callback races under + sanitizers where available. +- Every supported OS, architecture, libc/deployment floor, and language adapter + variant at the artifact level. diff --git a/bindings/docs/dynamic-extensions/nodejs.md b/bindings/docs/dynamic-extensions/nodejs.md new file mode 100644 index 000000000000..91e39a4a1aed --- /dev/null +++ b/bindings/docs/dynamic-extensions/nodejs.md @@ -0,0 +1,384 @@ + + +# Node.js Dynamic Extension Design + +Status: pre-RFC binding-specific design proposal. The current Node.js binding +remains one native addon with compile-time service and layer selection. + +The exact-build extension design is useful for Node.js. Node-API makes the base +addon portable across supported Node.js versions, but it does not stabilize the +Rust interface between separately built OpenDAL native packages. + +This document applies the [shared extension architecture](README.md) with +Node-specific package loading, addon initialization, Worker, and event-loop +constraints. +The [shared compatibility contract](compatibility.md) is canonical for native +ABI, configuration, lifetime, and loader rules; this document defines Node.js +deltas. + +## Current Constraints + +- `bindings/nodejs` builds one napi-rs `cdylib` using Node-API version 6. +- Services are selected by Cargo features. The default includes S3 and WebDAV; + the published feature set has explicit exclusions and target differences. +- `Operator` owns both asynchronous and blocking OpenDAL operators. +- Current layers are binding-local `NodeLayer` trait objects wrapped in + napi-rs `External` values. +- An `External` from one independently built addon is not a supported handle + for another addon. Its Rust type and layout belong to the addon that created + it. +- The generated loader already selects target packages by operating system, + architecture, and Linux libc, but it loads only one monolithic `.node` file. +- The package currently exposes only its root and `package.json` subpaths. +- ESM and CommonJS wrappers maintain central layer export lists and must stay in + sync. + +The dynamic design should reuse the target-package pattern without exposing +napi-rs implementation types as the extension contract. + +## npm Package Layout + +The proposed family is: + +```text +@opendal/runtime Node-API adapter and JavaScript interface +@opendal/runtime-linux-x64-gnu target-specific runtime addon +@opendal/service-s3 S3 JavaScript stub and types +@opendal/service-s3-linux-x64-gnu target-specific S3 native addon/library +@opendal/layer-timeout Timeout stub and types +@opendal/layer-foyer Foyer stub and types +opendal compatibility aggregator and re-export +``` + +Each root extension package declares: + +- An exact peer dependency on the compatible `@opendal/runtime` release. +- Target packages as optional dependencies with `os`, `cpu`, and `libc` + metadata where available. +- ESM and CommonJS entry points that use one registration implementation. +- A mandatory embedded runtime build-ID check. +- A clear error for optional dependencies omitted during installation. + +The unscoped `opendal` package preserves the current import and installs the +runtime plus the service/layer set provided by the current release. It re-exports +the runtime classes and registers those packages' text manifests. It does not +contain a second native runtime. + +Minimal applications depend directly on `@opendal/runtime` and selected +extensions. Package names remain provisional pending npm namespace and release +prototypes. + +One root package plus several target packages per extension creates a large +publication matrix. A registry with 1,000 manifest records can be efficient; +that does not prove that publishing or loading 1,000 native npm package families +is operationally practical. + +## Runtime Ownership + +The native architecture gives process resources and Node environment state +different owners: + +```text +loaded @opendal/runtime native module + ProcessRuntime + OpenDAL core, Tokio, registries, activation state, handle identity + process-lifetime native library leases + +main napi_env Worker napi_env + EnvironmentAdapter A EnvironmentAdapter B + JS wrappers and callbacks JS wrappers and callbacks + Promise completion bridge Promise completion bridge +``` + +One loaded runtime native module owns one `ProcessRuntime`. The exact peer +dependency should normally produce one such module in a process; nested +incompatible runtime installations can load distinct modules with distinct +process-runtime identities. The `ProcessRuntime` owns manifest conflicts, +activate-once state, native factories, Tokio/core resources, native handles, and +library leases. + +Every `napi_env` owns one `EnvironmentAdapter`. It owns JavaScript wrappers, +references, resolver/activation callbacks, Promise completion bridges, and +environment cleanup hooks. Registering through an environment adapter commits +the manifest atomically into its `ProcessRuntime`; an identical process-level +registration is idempotent and a different owner is a conflict. Native package +activation runs once per `ProcessRuntime`, not once per Worker. + +Node-API environments can be initialized and destroyed multiple times and can +run concurrently in Workers. A `napi_env`, `napi_value`, reference, JavaScript +callback, or public wrapper must never move between environment adapters. Node +documents these environment-lifecycle rules in its [Node-API +documentation](https://nodejs.org/api/n-api.html). + +Environment cleanup stops new calls, cancels or finishes environment-owned +asynchronous work, releases all JavaScript references, and removes its local +callbacks. It does not unload process-pinned extension libraries or invalidate +native handles owned by another environment. A Worker can therefore terminate +independently of operations in another Worker. + +## Native Registration Adapter + +The first Node prototype should compare two packaging implementations: + +1. The `ProcessRuntime` loads a language-neutral native library directly from + the package manifest. +2. An environment-bound JavaScript activator loads a target-specific Node-API + descriptor addon in the calling `napi_env`. + +Both implementations use the same descriptor, runtime build ID, factories, and +conformance suite. A literal separately installed shared Rust `dylib` is not a +design assumption; npm/pnpm/Yarn layouts, rpaths, and Windows DLL discovery must +prove it first. + +Direct host loading is the preferred starting point because it naturally +preserves text registration, process-level activate-once behavior, and +construction-time native activation. + +The Node-API descriptor variant requires this explicit environment-bound +activation sequence: + +1. The JavaScript registration stub gives its `EnvironmentAdapter` a text + manifest and a local activation callback. It loads no native target package. +2. First construction asks the adapter, on its JavaScript thread, to invoke that + callback. +3. The callback resolves the package's target artifact and synchronously loads + its `.node` addon in the same `napi_env`. ESM and CommonJS wrappers call one + shared loader implementation. +4. The addon uses the normal Node-API initializer and returns a `napi_external` + containing the C-layout descriptor. It does not return a napi-rs class or an + `External` shared with the base. +5. The base validates the descriptor before giving its language-neutral exact + entry to the `ProcessRuntime`. The process runtime installs the factories + atomically and records an explicit process-lifetime library lease. +6. The environment adapter releases its activation callback after success. The + factory does not retain `napi_env`, `napi_value`, JavaScript references, or + thread-safe functions. + +The variant is viable only if the platform prototype can retain a native +library lease independently from the initiating environment. A variant that +loads the HDFS addon during registration, or whose callbacks become invalid +when the initiating Worker exits, fails the design requirements. + +The base schedules extension futures on runtime-owned native resources and +adapts completion into the calling environment. + +## Registration Interface + +Explicit registration avoids bundler-dependent side effects: + +```javascript +import { Operator } from "@opendal/runtime"; +import { registerS3 } from "@opendal/service-s3"; + +registerS3(); + +const op = Operator.fromUri("s3://photos/archive", { + region: "us-east-1", +}); +``` + +`registerS3()` is idempotent for the same package and `ProcessRuntime`. It +registers a text manifest without activating native code. Under the descriptor +addon prototype it also installs one environment-local activation callback; +under direct host loading the manifest's artifact path is sufficient. + +The compatibility `opendal` aggregator can register its dependency set during +module initialization so existing construction remains concise: + +```javascript +import { Operator } from "opendal"; + +const op = new Operator("s3", { + bucket: "photos", + region: "us-east-1", +}); +``` + +The aggregator imports registration stubs, not every native target library. +Native activation still occurs on first construction. + +A package may offer a documented side-effect registration subpath for +convenience, but it must mark that subpath appropriately for bundlers. The +explicit function remains the unambiguous interface. + +## Proposed Layer Interface + +`Operator.layer()` should accept a base-owned opaque `Layer`, not +`ExternalObject` from napi-rs internals. + +Synchronous factories work for layers without asynchronous resources: + +```javascript +import { TimeoutLayer } from "@opendal/layer-timeout"; +import { ThrottleLayer } from "@opendal/layer-throttle"; + +const timeout = new TimeoutLayer(); +timeout.timeout = 60_000; +timeout.ioTimeout = 10_000; + +const limit = new ThrottleLayer(10 * 1024, 10 * 1024 * 1024); + +const layered = op.layer(limit.build()).layer(timeout.build()); +``` + +The compatibility `opendal` package preserves these current constructors, +setters, and `.build()` calls. Internally, `.build()` returns a base-owned opaque +`Layer` instead of a package-local napi-rs `External`. If the generated +`ExternalObject` TypeScript name cannot remain as a deprecated alias, the +type-name change must wait for the binding's next breaking public release; it +does not justify weakening the native handle boundary. + +Foyer uses a Promise-returning factory because JavaScript constructors cannot +be asynchronous: + +```javascript +import { FoyerLayer } from "@opendal/layer-foyer"; + +const cache = await FoyerLayer.create({ + memoryCapacity: 64 << 20, + storagePath: "/var/cache/opendal", +}); + +const cached = op.layer(cache); +``` + +The binding should not provide a synchronous Foyer constructor that blocks the +event loop. A separately documented worker/off-thread helper can be evaluated +later. + +Layer packages validate JavaScript numbers before calling Rust. Timeout values +use non-negative integer milliseconds and convert to the shared +[`DurationNs`](compatibility.md#configuration-value-contract) representation. +The adapter rejects values above `u64::MAX` nanoseconds and never truncates, +saturates, or wraps them. Throttle bandwidth and burst must be positive integers +in the supported `u32` range so invalid input cannot reach the core +constructor's assertions. Other options convert through the same shared +`ConfigValueV1` grammar; native factories never receive JavaScript objects. + +Applying one Throttle or Foyer handle to several operators preserves shared +native state. Later `.layer()` calls remain outer layers. Timeout must be +applied before Retry so each attempt has a deadline. Applying an outer Timeout +after Retry is unsupported because it can cancel Retry before operation-body +state is restored. The adapter preserves order and rejects this known unsafe +composition when both layer IDs are visible. + +## Async Operations and Cancellation + +The base adapter owns the conversion between runtime futures and JavaScript +Promises. Extension factories and operations do not call Node-API from Tokio +worker threads. + +Promise cancellation policy must be explicit because JavaScript Promises do not +provide universal cancellation. Where an operation accepts an `AbortSignal`, +the adapter forwards it into the runtime and drops or aborts the native future +according to OpenDAL semantics. + +Worker termination must not leave a callback targeting a destroyed `napi_env`. +Process-level native work may outlive one environment only when it has no +environment-owned completion callback and its resources have an explicit owner. + +## ESM, CommonJS, and Bundlers + +- ESM and CommonJS exports converge on one `EnvironmentAdapter` per Node + environment and the same `ProcessRuntime` for that loaded runtime module. +- Calling registration through both module systems is idempotent. +- Export maps include a supported registration/runtime subpath instead of + relying on generated private files. +- Side-effect-only registration modules declare their side effects so bundlers + do not remove them. +- Package stubs resolve native artifacts relative to their own installed + package, not the current working directory. +- Errors distinguish an unsupported target from installation with + `--omit=optional` and from an incompatible runtime. + +Dynamic native extensions are scoped to native Node-API targets. WASI and other +environments without compatible dynamic loading require a static bundled +design and should not silently fall back to this interface. + +## Version and Error Behavior + +Node-API version compatibility and OpenDAL runtime compatibility remain +separate. The generated package-version checks used by a native loader are not +a replacement for a mandatory embedded build-ID handshake. + +Extension lifecycle errors should be JavaScript `Error` subclasses or errors +with stable codes: + +```text +OPENDAL_EXTENSION_NOT_INSTALLED +OPENDAL_EXTENSION_LOAD_FAILED +OPENDAL_EXTENSION_INCOMPATIBLE +OPENDAL_EXTENSION_CONFLICT +OPENDAL_LAYER_INITIALIZATION_FAILED +``` + +Errors retain package ID, scheme/layer ID, target, and construction operation. +They do not expose credentials or an unredacted option object. Normal OpenDAL +errors retain their structured kind instead of becoming only a formatted +reason string. + +## Multiple Runtime Versions + +npm can install nested copies of a package. An extension stub may therefore see +a different `@opendal/runtime` instance from the one that created an operator. + +The design applies three defenses: + +1. Exact peer dependencies make the intended singleton visible to the package + manager. +2. Registration records the specific `ProcessRuntime` identity. +3. A layer/operator wrapper verifies process-runtime identity before native + handle use and throws `OPENDAL_EXTENSION_INCOMPATIBLE` on mismatch. + +The adapter must never reinterpret a handle from another `ProcessRuntime`, even +when package versions appear equal. JavaScript wrappers also remain confined to +their creating `EnvironmentAdapter`. + +## Migration + +1. Introduce `ProcessRuntime`, `EnvironmentAdapter`, and the extension registry + inside the current addon. +2. Replace the public `ExternalObject` detail with a base-owned opaque + layer wrapper while retaining current constructors. +3. Make compiled services/layers use the internal extension factory model. +4. Publish runtime and target packages using the existing platform-loader + experience. +5. Extract S3 and Timeout as tracer package families and register them from the + `opendal` compatibility aggregator. +6. Add Foyer async creation and HDFS lazy activation as design gates. +7. Test both direct native loading and Node-API descriptor addons before + selecting the physical linking model. +8. Publish the third-party SDK only after Worker, ESM/CommonJS, target, and + lifetime conformance passes. + +## Node.js Conformance Gates + +- ESM/CommonJS double registration, one environment adapter per `napi_env`, and + process-level activate-once behavior. +- Main thread plus multiple Workers importing, using, and terminating adapters. +- Two incompatible nested runtime versions rejecting cross-instance handles. +- Missing optional target package and unsupported target diagnostics. +- glibc/musl, macOS, and Windows artifact selection. +- Bundler retention for any documented side-effect registration entry. +- S3/WebDAV construction without central JavaScript config schemas. +- HDFS registration without Java/Hadoop and isolated activation failure. +- Foyer Promise construction, rejection, cleanup, and reusable handle state. +- Timeout executor behavior and Throttle shared identity through extracted + packages. +- No callback into a destroyed Node environment or unloaded native library. +- 1,000 synthetic registrations without loading 1,000 native addons. diff --git a/bindings/docs/dynamic-extensions/python.md b/bindings/docs/dynamic-extensions/python.md new file mode 100644 index 000000000000..a76416d56101 --- /dev/null +++ b/bindings/docs/dynamic-extensions/python.md @@ -0,0 +1,356 @@ + + +# Python Dynamic Extension Design + +Status: pre-RFC binding-specific design proposal. The current Python binding +remains a single native distribution. + +This document applies the [shared extension architecture](README.md) to Python. +It preserves `Operator` and `AsyncOperator` while moving service and layer +implementation dependencies into independently installable distributions. +The [shared compatibility contract](compatibility.md) is canonical for native +ABI, configuration, lifetime, and loader rules; this document defines Python +deltas. + +## Current Constraints + +The current Python binding has several compile-time assumptions that cannot +serve as a dynamic extension interface: + +- `bindings/python` builds one PyO3 `cdylib`, `opendal._opendal`. +- Published wheels enable the binding's `services-all` feature, subject to its + explicit exclusions and platform conditions. +- `Scheme` is a feature-gated Rust enum. It cannot represent a service installed + after the base extension was compiled. +- `opendal.services` and `opendal.layers` are PyO3 submodules inserted into + `sys.modules`, not filesystem packages that other distributions can extend. +- `Layer` stores `Box` in the base native library. A PyO3 + subclass marker does not make that Rust trait object transferable from an + independently linked extension. +- `opendal.config.ServiceConfig` is one generated closed union of compiled + services. +- Operator pickle state records construction URI/options but does not record + applied layers. + +The migration must change these internals without requiring every caller to +adopt a new operator abstraction. + +## Distribution Layout + +The proposed release family is: + +```text +opendal-runtime owns the `opendal` import package and native runtime +opendal-service-s3 contributes the S3 manifest, native code, and typing +opendal-service-hdfs contributes libhdfs-backed HDFS lazily +opendal-layer-timeout contributes Timeout +opendal-layer-foyer contributes Foyer +opendal compatibility aggregator with no native extension +``` + +Every native service/layer distribution requires an exact +`opendal-runtime` release and embeds its runtime build ID. The `opendal` +aggregator depends on the runtime and the service/layer set shipped by the +current monolithic `opendal` wheel. This preserves: + +```console +python -m pip install opendal +``` + +A minimal application instead installs only the runtime and selected packages: + +```console +python -m pip install \ + opendal-runtime \ + opendal-service-s3 \ + opendal-layer-timeout \ + opendal-layer-foyer +``` + +The package names are provisional. The split between runtime and aggregator is +not: allowing both distributions to install the same native `opendal` files +would create ambiguous ownership and upgrades. + +## Import Layout + +The intended typed import layout is: + +```text +opendal regular package owned by opendal-runtime +opendal.services namespace subpackage +opendal.services.s3 supplied by opendal-service-s3 +opendal.services.hdfs supplied by opendal-service-hdfs +opendal.layers namespace subpackage +opendal.layers.timeout supplied by opendal-layer-timeout +opendal.layers.foyer supplied by opendal-layer-foyer +``` + +Python packaging supports splitting namespace subpackages across +distributions, but every participant must follow one consistent layout. See +the [PyPA namespace package guide](https://packaging.python.org/en/latest/guides/packaging-namespace-packages/). + +Before using this layout, the base binding must move the current native +`opendal.services` and `opendal.layers` definitions under a private native +module and expose real Python package directories. Existing flat names can be +re-exported during migration. + +If wheel-install and namespace ownership prototypes are not reliable across the +supported installers, the first tracer packages may use unambiguous top-level +imports such as `opendal_service_s3`. The runtime extension contract does not +depend on the cosmetic import layout. + +## Discovery and Activation + +Python entry points advertise installed manifests: + +```toml +[project.entry-points."opendal.services"] +s3 = "opendal.services.s3:_register" + +[project.entry-points."opendal.layers"] +foyer = "opendal.layers.foyer:_register" +``` + +[Entry points](https://packaging.python.org/en/latest/specifications/entry-points/) +allow the runtime to find an installed package without importing every package. +The resolver follows these rules: + +1. `import opendal` loads only the base adapter and runtime. +2. An explicit service/layer import registers its text manifest. +3. Construction of an unregistered scheme must look up the one entry point with + the matching canonical name and load only that registration stub. It reports + a conflict if more than one distribution claims the name. +4. Native code activates only when the caller constructs that service/layer. +5. Resolver results and deterministic failures are cached. +6. Discovery never installs a missing distribution at runtime. + +Explicit imports remain useful for deterministic startup and access to typed +configuration classes. Entry-point discovery preserves the current concise URI +path for callers that only need strings. This lookup is mandatory: dependencies +installed by the compatibility `opendal` aggregator do not register themselves +merely because pip installed them. + +## Proposed Operator Interface + +Existing construction remains valid: + +```python +import opendal + +op = opendal.Operator("s3", bucket="photos", region="us-east-1") + +async_op = opendal.AsyncOperator.from_uri( + "s3://photos/archive?region=us-east-1", + endpoint="https://s3.example.com", +) +``` + +Strings are the canonical dynamic scheme identifiers. The existing `Scheme` +enum may remain as a frozen compatibility aid for previously bundled official +services, but it is not an inventory of installed extensions. + +Typed configuration moves into its service distribution: + +```python +from opendal import AsyncOperator +from opendal.services.s3 import S3Config + +config = S3Config(scheme="s3", bucket="photos", region="us-east-1") +op = AsyncOperator.from_config(config) +``` + +The base `from_config` runtime path accepts a generic mapping or service recipe. +Package-local generated `TypedDict` or dataclass definitions provide field +checking without extending one base `ServiceConfig` union. The service package +owns structured serialization and schema revisioning. + +The adapter converts mappings to the shared +[`ConfigValueV1`](compatibility.md#configuration-value-contract) grammar. It +rejects unsupported Python objects, cyclic containers, oversized values, +unknown fields, and numeric overflow before package construction. Package-local +types can expose Python-native values, but no `PyObject` crosses the factory +seam. + +URI construction sends the original URI plus explicit string options to the +service factory. It does not convert them through a central Python config +schema. This preserves S3 and WebDAV configurator behavior. + +## Proposed Layer Interface + +Simple layer factories remain synchronous: + +```python +from opendal.layers.throttle import ThrottleLayer +from opendal.layers.timeout import TimeoutLayer + +limit = ThrottleLayer(bandwidth=10 * 1024, burst=10 * 1024 * 1024) +timeout = TimeoutLayer(timeout=60.0, io_timeout=10.0) + +layered = op.layer(limit).layer(timeout) +``` + +Resource-backed construction is asynchronous: + +```python +from opendal.layers.foyer import FoyerLayer + +cache = await FoyerLayer.create( + memory_capacity=64 << 20, + storage_path="/var/cache/opendal", +) + +cached = async_op.layer(cache) +``` + +A blocking helper may be provided for synchronous applications only if it uses +the same runtime factory, releases the GIL while waiting, and has defined +cancellation/cleanup behavior. It must not create a second Tokio runtime inside +the Foyer package. + +Every concrete Python layer wraps a base-owned opaque `LayerHandle`. It does +not expose a package-local Rust trait object. Applying it returns a new operator +and preserves the native layer's service and context hooks. + +One layer object may carry shared state: + +- Applying one Throttle object to two operators shares its quota. +- Applying two independently constructed Throttle objects creates two quotas. +- Applying one Foyer object reuses one cache, subject to the cache namespace + restriction documented by that package. + +Throttle accepts only positive integer `bandwidth` and `burst` values in the +supported `u32` range. Python validation must reject invalid values before the +native constructor can assert. + +Timeout values remain finite, non-negative seconds at the Python interface. The +adapter uses the current `Duration::try_from_secs_f64` rule, which rounds to the +nearest nanosecond with ties to even, and then emits `DurationNs`. It rejects +values above `u64::MAX` nanoseconds instead of saturating them. + +Later `.layer()` calls are outer layers. Timeout must be applied before Retry, +which puts Retry outside Timeout and gives each attempt a deadline. Applying +Timeout after Retry is unsupported because the outer Timeout can cancel Retry +before it restores operation-body state. The binding preserves order and +rejects that known unsafe composition when it can observe both layer IDs. + +## Async Behavior + +The shared runtime owns operation futures. `AsyncOperator` converts them into +Python awaitables through the base adapter. A service or layer package does not +capture Python event loops, `PyObject` references, or PyO3 runtime state in its +native factory. + +Cancellation of a Python awaitable must reach the runtime operation. The +adapter must not detach a future merely because its Python wrapper was dropped. + +Blocking `Operator` and `AsyncOperator` should retain the same constructed +native operator graph when converted or cloned. Rebuilding from a scheme and +options would lose stateful Foyer and Throttle identity. + +## Errors + +The Python adapter should distinguish extension failures before mapping normal +OpenDAL operation errors: + +```text +ExtensionNotInstalled +ExtensionLoadError +ExtensionIncompatible +ExtensionConflict +LayerInitializationError +``` + +`ExtensionNotInstalled` may include the canonical distribution name as an +installation hint. It must not run `pip`, modify the environment, or infer a +third-party package name from untrusted input. + +Configuration and operation errors continue to use OpenDAL's Python exception +hierarchy. Extension diagnostics include package and scheme/layer IDs but omit +credentials and unredacted option maps. + +## Serialization + +A dynamic operator cannot rely on native pointer serialization. New pickle +support must choose one of these explicit policies: + +- Serialize a versioned service recipe plus ordered layer recipes, then + reconstruct fresh native resources. +- Reject pickling when an applied service/layer has no declarative + reconstruction policy. + +It must never silently discard layers. Reconstructing a Foyer layer creates or +reopens a cache according to package policy; it does not preserve live in-memory +entries. Reconstructing Throttle starts new token history. + +Existing layered pickles did not record layer recipes, so migration code cannot +recover that lost information retroactively. + +## Packaging Constraints + +The current release matrix includes CPython 3.10-specific wheels, CPython 3.11 +`abi3` wheels, and free-threaded CPython wheels. The dynamic design must prove +which artifacts can actually be shared: + +- The Python adapter follows its existing CPython/`abi3` compatibility rules. +- A language-neutral service/layer library should not link CPython. +- The extension still needs one artifact per supported native target and libc + or deployment floor. +- One extension wheel can cover several Python versions only if all of those base + wheels load the same runtime build ID and compatible shared library. +- Wheel repair must retain the intended shared runtime relationship instead of + copying private runtime libraries into every extension under conflicting + names. +- Free-threaded Python requires explicit lifetime and concurrency tests; `abi3` + does not imply free-threaded compatibility. + +The libhdfs-backed HDFS wheel may have a smaller platform allowlist or ship as a +source distribution. Installing `opendal-runtime`, S3, WebDAV, or +`hdfs-native` must not load HDFS code or require Java/Hadoop. + +## Migration + +1. Add the runtime, descriptor, and registry internally while services/layers + remain compiled into the base wheel. +2. Make built-in adapters use the same internal factory interface intended for + external packages. +3. Turn `opendal.services` and `opendal.layers` into filesystem/namespace + packages and re-export existing names. +4. Extract S3 and Timeout as tracer distributions. Keep Memory in the runtime + because OpenDAL core always provides it. +5. Convert `opendal` into the compatibility aggregator before removing bundled + features from its runtime dependency. +6. Generate package-local configuration types from the Rust service metadata. +7. Validate WebDAV, HDFS, Foyer, and Throttle before declaring the interface + complete. +8. Introduce versioned serialization or explicit non-picklability for dynamic + operators. + +## Python Conformance Gates + +- Base import with no optional extensions installed. +- Explicit imports and lazy entry-point lookup with 1,000 synthetic manifests. +- Namespace-package coexistence under pip and other supported installers. +- Sync and async S3/WebDAV construction through URI and typed config paths. +- HDFS registration/import without native activation and scoped load failure. +- Foyer async creation, cancellation, garbage collection, and reuse. +- Timeout executor behavior through a separately packaged layer. +- Throttle argument validation and shared-handle identity. +- Pickle reconstruction or explicit rejection with missing/incompatible + packages. +- CPython 3.10, `abi3`, and free-threaded artifact composition on every + supported target. diff --git a/bindings/docs/dynamic-extensions/ruby.md b/bindings/docs/dynamic-extensions/ruby.md new file mode 100644 index 000000000000..905ec78bb853 --- /dev/null +++ b/bindings/docs/dynamic-extensions/ruby.md @@ -0,0 +1,331 @@ + + +# Ruby Dynamic Extension Design + +Status: pre-RFC binding-specific design proposal. The current Ruby binding +remains one gem and one Magnus native extension. + +This document applies the [shared extension architecture](README.md) to Ruby. +It incorporates the version-locked shared runtime alternative from the earlier +Ruby design and aligns that native model with Python and Node.js. +The [shared compatibility contract](compatibility.md) is canonical for native +ABI, configuration, lifetime, and loader rules; this document defines Ruby +deltas. + +## Current Constraints + +The current Ruby binding provides a useful migration base but not a native +extension seam: + +- `bindings/ruby` builds one `opendal_ruby` `cdylib` and one `opendal` gem. +- `OpenDal::Operator.new(scheme, options)` is blocking-only and constructs + through the compiled core registry. +- Ruby does not currently expose `Operator.from_uri`, `Operator.via_iter`, or an + async operator. +- Retry, concurrent-limit, Throttle, and Timeout middleware implementations are + compiled into the same native extension. +- `Operator#middleware` uses Ruby duck typing, but an independently built native + middleware still cannot access the wrapped Rust `Operator` in another DSO. +- Operation failures currently map broadly to Ruby `RuntimeError`. +- The release process builds a source gem and a small best-effort native-gem + matrix. A source-build path remains important. + +The dynamic design must not describe proposed methods or guarantees as current +behavior. + +## Gem Layout + +The proposed release family is: + +```text +opendal-runtime owns `require "opendal"` and the shared runtime +opendal-service-s3 provides S3 registration and native artifacts +opendal-service-hdfs provides libhdfs-backed HDFS lazily +opendal-layer-timeout provides Timeout +opendal-layer-foyer provides Foyer +opendal compatibility aggregator +``` + +Each native service/layer gem requires an exact `opendal-runtime` release and +embeds its runtime build ID. The `opendal` compatibility gem depends on the +runtime and the extensions included by the current monolithic gem. This +preserves: + +```console +gem install opendal +``` + +Applications that need a smaller installation select the runtime and extensions +in their `Gemfile`: + +```ruby +gem "opendal-runtime", "= " +gem "opendal-service-s3" +gem "opendal-layer-timeout" +gem "opendal-layer-foyer" +``` + +The aggregator must not install files that conflict with the runtime gem. It is +metadata and dependency coordination, not a second owner of `lib/opendal.rb` or +the runtime native library. + +## Registration and Activation + +Each extension gem contains a Ruby registration stub, a text manifest, and gem +metadata mapping its canonical service/layer IDs to that stub: + +```ruby +require "opendal/runtime" + +OpenDal::Runtime.register_manifest( + File.expand_path("../../../opendal-extension.json", __dir__) +) +``` + +The expected require paths are: + +```ruby +require "opendal" +require "opendal/services/s3" +require "opendal/layers/timeout" +require "opendal/layers/foyer" +``` + +Requiring an extension reads and registers metadata but does not activate its +native library. The first service/layer construction performs native loading +and the exact build-ID check. + +The compatibility aggregator publishes a generated index of the registration +stubs provided by its exact dependencies. When `require "opendal"` activates +that aggregator, the runtime requires each indexed registration stub. Those +stubs register text manifests only; they do not load native libraries. + +For a minimal installation, construction of an unregistered scheme must resolve +the one matching registration stub from installed gem metadata. The resolver +reports duplicate claims, caches results and deterministic failures, handles +aliases deterministically, and never requires every native extension at +startup. Installing dependencies alone is not treated as registration. + +Explicit `require` remains the preferred deterministic registration path. An +application that wants to detect native dependency failures during controlled +startup must also construct or explicitly probe the service/layer, because +registration alone intentionally performs no native load. + +## Proposed Operator Interface + +`Operator.new` remains the compatibility constructor: + +```ruby +require "opendal" +require "opendal/services/s3" + +op = OpenDal::Operator.new("s3", { + "bucket" => "photos", + "region" => "us-east-1", +}) +``` + +The binding can add URI and explicit registry construction as additive methods: + +```ruby +op = OpenDal::Operator.via_iter("s3", { + "bucket" => "photos", + "region" => "us-east-1", +}) + +op = OpenDal::Operator.from_uri( + "s3://photos/archive?region=us-east-1", + {"endpoint" => "https://s3.example.com"} +) +``` + +After those methods exist, `Operator.new` delegates to `via_iter`. Scheme +strings remain canonical so third-party services do not require edits to a +base enum. + +The service gem receives the original URI and explicit string options. S3, +WebDAV, HDFS, and third-party gems retain their own configurator behavior, +validation, credentials, and redaction. + +Typed Ruby configuration objects and hashes convert to the shared +[`ConfigValueV1`](compatibility.md#configuration-value-contract) grammar. The +base adapter rejects symbols or objects without a declared conversion, cyclic +containers, oversized values, unknown fields, and numeric overflow before +calling package code. Native factories never retain Ruby objects. + +## Proposed Layer Interface + +New code uses `OpenDal::Layers` and `Operator#layer`: + +```ruby +require "opendal/layers/throttle" +require "opendal/layers/timeout" + +limit = OpenDal::Layers::Throttle.new(10 * 1024, 10 * 1024 * 1024) +timeout = OpenDal::Layers::Timeout.new(60, 10) + +layered = op.layer(limit).layer(timeout) +``` + +The Ruby object wraps a runtime-owned native `LayerHandle`, not a package-local +Rust object exposed through Magnus. `Operator#layer` returns a new operator and +preserves native service and context hooks. + +The current names remain compatibility adapters: + +- `Operator#middleware(value)` delegates to `Operator#layer(value)`. +- `OpenDal::Middleware::*` aliases the corresponding `OpenDal::Layers::*` + classes during a deprecation period. The layer classes preserve the current + positional constructors so the aliases do not change existing calls. +- A pure Ruby object implementing only `apply_to` remains a Ruby decorator and + must not be described as equivalent to an arbitrary native layer. + +Timeout values remain finite, non-negative seconds at the Ruby interface. The +adapter uses the current `Duration::try_from_secs_f64` rule, which rounds to the +nearest nanosecond with ties to even, and then emits `DurationNs`. It rejects +values above `u64::MAX` nanoseconds instead of saturating them. + +Throttle accepts only positive integer `bandwidth` and `burst` values in the +supported `u32` range. Ruby validation must reject invalid values before the +native constructor can assert. + +Later layer calls are outer layers. Timeout must be applied before Retry, which +puts Retry outside Timeout and gives each attempt a deadline. Applying Timeout +after Retry is unsupported because the outer Timeout can cancel Retry before it +restores operation-body state. The binding preserves order and rejects that +known unsafe composition when it can observe both layer IDs. + +## Asynchronous Layer Construction + +Ruby operations remain blocking in the initial design. A layer such as Foyer +still needs asynchronous native initialization: + +```ruby +require "opendal/layers/foyer" + +cache = OpenDal::Layers::Foyer.build( + memory_capacity: 64 << 20, + storage_path: "/var/cache/opendal" +) + +cached = op.layer(cache) +``` + +`Foyer.build` submits the async factory to the shared runtime and waits while +releasing the GVL. It must use the runtime's Tokio instance, clean up partial +resources on failure, and return only after it owns a valid `LayerHandle`. + +The package must not start a private Tokio runtime or hold Ruby values inside +its native future. A future Ruby async interface can adapt the same runtime +future without changing the extension interface. + +## Stateful Layers + +One Ruby layer object preserves one native sharing identity: + +- Applying one Throttle object to several operators shares one quota. +- Constructing two Throttle objects creates independent quotas. +- Applying one Foyer object shares one cache subject to its documented + namespace restriction. +- Derived operators keep the layer alive after the original Ruby wrapper is + collected. + +The first version does not define `Marshal` support for operators or live layer +handles. A future declarative recipe format must reconstruct new native state +rather than claiming to serialize cache contents, limiter history, a JVM, or a +Tokio runtime. + +## Errors + +The runtime should expose Ruby exception classes for extension lifecycle +failures: + +```text +OpenDal::ExtensionNotInstalled +OpenDal::ExtensionLoadError +OpenDal::ExtensionIncompatible +OpenDal::ExtensionConflict +OpenDal::LayerInitializationError +``` + +Normal OpenDAL error kinds should also map to stable Ruby exception classes +rather than losing all structure in `RuntimeError`. Compatibility aliases or a +common superclass can preserve existing rescue behavior. + +Errors include package ID, scheme/layer ID, and construction operation. They do +not include credentials or unredacted option hashes. + +## Native Gem and Loader Constraints + +- The base Magnus extension follows the binding's supported Ruby versions and + platforms. +- A language-neutral service/layer library should not link Ruby or Magnus. +- Native extension gems still need artifacts for every supported OS, + architecture, libc/deployment floor, and runtime build ID. +- Source gems build against the exact SDK/runtime metadata and verify the + resulting embedded build ID. +- The runtime loads an explicit artifact path from the gem manifest and keeps + the library pinned. +- Linux symbol visibility, macOS install names, and Windows DLL discovery must + be tested with gems installed in normal Bundler layouts. +- A native-gem failure may fall back to a documented source build, but it must + not silently load a different runtime build. + +The current native-gem matrix is best effort. Dynamic extensions should not +claim broader binary coverage until runtime plus adapter artifacts pass an +installation test on that platform. + +## Ractor, Threads, and Fork + +The first design does not promise Ractor shareability. Runtime registries and +native handles may be process-global Rust state, but they must not retain +Ractor-local Ruby objects. + +Blocking operations and layer initialization release the GVL only through +well-defined base-adapter helpers. Package code must not call Ruby from shared +runtime worker threads. + +Runtime, JVM, connection-pool, and Foyer state is unsupported after `fork` +unless a package later defines explicit reinitialization behavior. + +## Migration + +1. Introduce `NativeRuntime` and an extension registry inside the current gem. +2. Route compiled services and middleware through internal factories using the + proposed SDK shapes. +3. Add `via_iter`, `from_uri`, `layer`, `OpenDal::Layers`, and structured errors + without removing current methods. +4. Extract S3 and Timeout as tracer gems and make `opendal` depend on them. +5. Add gem-metadata resolution for compatibility constructors. +6. Validate WebDAV configurator behavior and HDFS lazy activation. +7. Validate Foyer initialization and Throttle sharing before publishing the + third-party SDK. +8. Expand native gem targets only after artifact-level installation tests pass. + +## Ruby Conformance Gates + +- Base runtime installation and `require "opendal"` without optional extensions. +- Explicit requires and selected lazy resolution without eager native loading. +- Existing `Operator.new` and `middleware` compatibility behavior. +- Proposed URI construction after `from_uri` is implemented. +- S3/WebDAV secret-free construction errors. +- HDFS failure scoped to the HDFS gem while other services remain usable. +- Foyer initialization with the GVL released, including cancellation/cleanup. +- Layer wrapper garbage collection while derived operators remain active. +- Source gem plus every claimed native gem target. +- Thread, Ractor-rejection, and fork-rejection behavior documented and tested. From fd06458992b7b26f6bc37cf8226d41c3ec336046 Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Sat, 1 Aug 2026 13:17:23 +0800 Subject: [PATCH 2/8] docs: address dynamic extension review feedback --- bindings/docs/dynamic-extensions/README.md | 90 +-- .../docs/dynamic-extensions/alternatives.md | 261 +++------ .../docs/dynamic-extensions/compatibility.md | 518 +++++++----------- bindings/docs/dynamic-extensions/nodejs.md | 99 ++-- bindings/docs/dynamic-extensions/python.md | 53 +- bindings/docs/dynamic-extensions/ruby.md | 55 +- 6 files changed, 392 insertions(+), 684 deletions(-) diff --git a/bindings/docs/dynamic-extensions/README.md b/bindings/docs/dynamic-extensions/README.md index 168274265c6b..57ed1ec0615e 100644 --- a/bindings/docs/dynamic-extensions/README.md +++ b/bindings/docs/dynamic-extensions/README.md @@ -24,16 +24,16 @@ This design allows a language binding to install services and layers as independent packages. For example, an application can install only S3, Timeout, and Foyer without rebuilding or replacing the base binding. -The leading prototype candidate uses an exact-build shared native runtime. +The leading prototype candidate uses an exact-release shared native runtime. Python, Ruby, and Node.js use different language interfaces over the same native extension model. Selecting that candidate remains conditional on a successful cross-platform packaging prototype and the OpenDAL RFC process. ## Documents -- [Design alternatives](alternatives.md) compares four possible extension +- [Design alternatives](alternatives.md) compares two possible extension seams and explains why Design C is the leading prototype candidate. -- [Compatibility and ABI](compatibility.md) defines version identities, +- [Compatibility and ABI](compatibility.md) defines version and target checks, loader checks, lifetime rules, and supported stability guarantees. - [Python design](python.md) defines Python packages, discovery, typing, and migration. @@ -53,11 +53,8 @@ The design must satisfy all of the following requirements: - A service or layer package can be installed without rebuilding the base language binding. -- The base binding loads without dependencies required by unselected packages. - In particular, HDFS must not make Java, Hadoop, `libhdfs`, or `libjvm` a base - requirement. - A service package owns its configuration schema, URI interpretation, - credentials, and redaction behavior. + credentials, redaction behavior, and registry metadata. - A layer package preserves the complete native `Layer` behavior, including both service and operation-context composition. - Stateful layers can be reused intentionally. Reusing one Throttle handle @@ -123,8 +120,30 @@ Python adapter Ruby adapter Node-API adapter The diagram shows a logical architecture. The first implementation may package a separate runtime artifact for each language ecosystem. Sharing the SDK, -descriptor, rules, and conformance suite is required; sharing one physical -runtime library across Python, Ruby, and Node.js installations is not. +manifest rules, and conformance suite is required; sharing one physical runtime +library across Python, Ruby, and Node.js installations is not. + +### Why one common runtime graph is required + +Services and layers from independent packages must compose into the same +language-level `Operator`. A native layer must be able to wrap an operator from +another package while preserving both `Layer::apply_service` and +`Layer::apply_context`. That requires shared OpenDAL types, Tokio resources, +HTTP and executor context, registry ownership, and handle identity. + +If each package owned a separate OpenDAL runtime graph, Rust `Operator`, `Layer`, +and `OperationContext` values could not cross package boundaries through a +supported stable ABI. Cross-package composition would then require either a +second stable operation interface or a complete operation adapter in each +language. The project does not plan to maintain those larger interfaces, and a +language adapter cannot preserve arbitrary native layer semantics. + +The common runtime graph therefore owns composition machinery and extensions +register release-specific factories into it. Service packages still own their +configuration, URI interpretation, credentials, redaction, and package-local +dependencies. "Common" means one graph for extensions loaded by one binding +runtime; it does not require one physical runtime library to be shared across +Python, Ruby, and Node.js. ### Binding adapter @@ -142,9 +161,9 @@ The adapter preserves each language's normal interface: streams, ESM/CommonJS exports, and Node-API. The project should share the native extension contract, not force one public -language API across all three bindings. The manifest, bootstrap descriptor, -configuration value grammar, factory semantics, error categories, and -conformance suite are common. Constructor names, typing, duration syntax, +language API across all three bindings. The JSON manifest and bootstrap +documents, configuration value grammar, factory semantics, error categories, +and conformance suite are common. Constructor names, typing, duration syntax, blocking behavior, and package discovery remain binding-specific. ### Extension runtime @@ -174,30 +193,32 @@ conflict-detecting registration. Each installable package contains: - Language code or type declarations. -- A text manifest with the package ID, provided schemes or layers, native - artifact path, unique entry symbol, and required runtime build ID. +- A JSON manifest for exactly one service or layer, with the package ID, + canonical component ID, service aliases when applicable, native artifact + path, unique entry symbol, and required OpenDAL version. - A target-specific native library, or source needed to build one. - Package-specific documentation and conformance tests. -Registering the text manifest must not load the native library. The runtime +Registering the JSON manifest must not load the native library. The runtime activates the library when a caller first constructs a declared service or layer. This isolates HDFS and similar native dependencies. ### Native extension -A native extension contains one or more service or layer factories. The -package exports one generated, package-unique bootstrap symbol and hides other +A native extension contains exactly one service or layer factory. The package +exports one generated, package-unique bootstrap symbol and hides other package-local symbols where the platform permits it. -The bootstrap uses a small C-layout descriptor so the runtime can inspect its -size, revision, manifest identity, target, and runtime build ID safely. Only -after all checks pass may the runtime enter the exact-build internal interface. -That internal interface is not a stable Rust ABI. +The bootstrap returns a bounded UTF-8 JSON document so the runtime can inspect +the package identity, target, and exact OpenDAL version through a small C +calling convention. Only after all checks pass may the runtime enter the +release-specific internal interface. That internal interface is not a stable +Rust ABI. Node.js packages are an exception to the unique-initializer rule when the native artifact is itself a Node-API addon: Node defines the addon initializer. The Node adapter must use a language-appropriate initializer and validate the -same package identity in the returned descriptor. +same package identity in the returned JSON document. ## Construction Requests @@ -210,12 +231,11 @@ UriRequest { } ConfigRequest { - package_schema_revision, structured_values, } ``` -`structured_values` uses the versioned, language-neutral `ConfigValue` grammar +`structured_values` uses the language-neutral `ConfigValue` grammar defined by [the compatibility contract](compatibility.md#configuration-value-contract). Each binding validates and converts its public values before invoking a native factory; a native package never receives Python, Ruby, or JavaScript objects. @@ -239,9 +259,11 @@ applications are outer layers, matching OpenDAL core behavior. ## Registration Rules - Scheme and layer IDs use lowercase canonical strings. -- One manifest can claim several names and aliases atomically. +- One manifest claims exactly one service or layer. A service manifest can also + declare aliases for its canonical scheme. - Re-registering the identical package and manifest is idempotent. -- A different owner claiming an existing name rejects the entire manifest. +- A different owner claiming the component ID or one of its aliases rejects the + manifest. - Registry locks are not held while loading code or running a factory. - Capabilities are read from a constructed operator, not declared statically in registration metadata. @@ -250,20 +272,16 @@ applications are outer layers, matching OpenDAL core behavior. ## Package Families -Each ecosystem needs a runtime distribution, independent extension packages, -and a compatibility aggregator: +Each ecosystem needs a runtime distribution and independent extension packages: ```text runtime distribution minimal binding and native runtime -service/layer packages exact dependency on that runtime release -compatibility aggregator runtime plus the services/layers shipped today +service/layer packages exact dependency on that OpenDAL release ``` -The aggregator preserves existing installation commands during migration. A -minimal installation uses the runtime distribution plus selected extension -packages. Independently installable does not imply independently -ABI-versioned: every exact-runtime native package must be rebuilt for a new -runtime build ID. +A minimal installation uses the runtime distribution plus selected extension +packages. Independently installable does not imply independently ABI-versioned: +every native package must be rebuilt for a new OpenDAL release. The 1,000-manifest requirement measures registry behavior, not a commitment to publish 1,000 official package families. Before a split release, each ecosystem @@ -277,7 +295,7 @@ Native extensions are trusted in-process code. Compatibility validation prevents accidental mismatches; it does not sandbox, authenticate, or constrain a malicious package. -The host still owns these in-scope properties: +OpenDAL still owns these in-scope properties: - No panic or language exception crosses any FFI boundary. - Safe binding operations do not cause use-after-free or data races. diff --git a/bindings/docs/dynamic-extensions/alternatives.md b/bindings/docs/dynamic-extensions/alternatives.md index 0e493e26cd12..6b9c05e2cc22 100644 --- a/bindings/docs/dynamic-extensions/alternatives.md +++ b/bindings/docs/dynamic-extensions/alternatives.md @@ -20,45 +20,43 @@ Status: pre-RFC design comparison. No alternative has been accepted or implemented. -This document compares four ways to deliver OpenDAL services and layers to -language bindings. The designs place the extension seam at different levels, -which changes their native fidelity, compatibility, and implementation cost. +This document compares the two implementation models that fit the project's +maintenance constraints. The designs differ in whether users select components +at build time or install them as packages later. ## Decision Guide ```text Must packages compose after installation? no -> Design A: feature-selected monolithic binding - yes -> Must arbitrary native layers preserve core semantics? - no -> Design B: language-owned operator adapters - yes -> Must an old binary survive runtime/core upgrades? - no -> Design C: exact-build shared runtime - yes -> Design D: stable C operation interface + yes -> Design C: exact-release shared runtime ``` OpenDAL should prototype Design C. Design A remains useful for custom builds. -Design B does not meet the native-layer requirement. Design D should be chosen -only if cross-version third-party binary compatibility becomes a strategic -requirement that justifies stabilizing a second operation model. +The project does not plan to maintain language-specific operation adapters or a +second stable operation ABI. ## Summary -| Property | A. Static build | B. Language adapters | C. Exact runtime | D. Stable C interface | -| ------------------------------ | ----------------- | -------------------- | ---------------------- | -------------------------- | -| Independently install services | No | Yes | Yes | Yes | -| Independently install layers | No | Language decorators | Yes | Yes | -| Full native layer semantics | Yes | No | Yes | Only represented semantics | -| One core/runtime graph | Yes | Usually no | Yes | Usually no | -| Third-party source extensions | Rebuild host | Yes | Yes, exact build | Yes | -| Cross-version native binaries | Not applicable | Language-dependent | No | Yes, within ABI rules | -| Cross-language native package | No | No | Possible, not promised | Yes, by design | -| Native dependency isolation | No | Yes | Yes | Yes | -| Implementation cost | Lowest | Moderate | High | Highest | -| Primary risk | Artifact variants | Semantic loss | Packaging/linking | ABI breadth and safety | + + +| Property | A. Static build | C. Exact-release runtime | +| ------------------------------ | ----------------- | ------------------------ | +| Independently install services | No | Yes | +| Independently install layers | No | Yes | +| Full native layer semantics | Yes | Yes | +| One core/runtime graph | Yes | Yes | +| Third-party source extensions | Rebuild host | Yes, exact release | +| Cross-version native binaries | Not applicable | No | +| Native dependency isolation | No | Yes | +| Implementation cost | Lowest | High | +| Primary risk | Artifact variants | Packaging and linking | + + ## Design A: Feature-Selected Monolithic Binding -### Seam +### Static Build Seam Cargo features select services and layers when the language extension is compiled. The resulting wheel, gem, or npm native package contains one OpenDAL @@ -68,86 +66,39 @@ core and all selected implementations. language caller -> one native binding -> compiled services and layers ``` -### Strengths +### Static Build Strengths - Preserves all native behavior without a new runtime interface. - Uses ordinary Rust ownership and one Tokio/core graph. - Has the smallest implementation and verification cost. - Works well for downstream users who build one controlled deployment image. -### Limitations +### Static Build Limitations - A service cannot be added after the binding is built. -- Two feature variants normally provide the same import/module name and cannot - be installed together. +- Two feature variants normally provide the same import or module name and + cannot be installed together. - One uncommon native dependency can constrain the whole artifact. A build - containing libhdfs-backed HDFS may fail to load where Java/Hadoop libraries + containing libhdfs-backed HDFS may fail to load where Java or Hadoop libraries are absent. - A large published feature set increases build time, artifact size, supply chain surface, and platform exclusions. -### Appropriate Use +### Static Build Use Keep this path for custom source builds and hermetic distributions. It does not meet the independently installable package requirement. -## Design B: Language-Owned Operator Adapters - -### Seam - -Each service package owns an operator and exposes a Python, Ruby, or JavaScript -operation protocol. The base binding delegates every operation through that -language protocol. Layers decorate language objects. - -```text -language Operator facade -> language operation protocol -> package-owned backend - ^ - | - language decorator -``` - -### Strengths - -- Package loading uses normal language mechanisms. -- Pure-language third-party services need no Rust ABI. -- Each native package can own its core version and runtime because Rust values - do not cross the seam. -- HDFS dependencies remain isolated in the HDFS package. - -### Limitations - -- Every service package repeats or generates the complete operation adapter. -- Language calls appear in streaming and operation-body paths unless the - adapter adds another native batching interface. -- A native OpenDAL layer cannot wrap an operator owned by another extension. -- Reproducing native layers eventually requires readers, writers, listers, - deleters, copiers, cancellation, HTTP context, and executor context. At that - point this design has recreated a language-specific version of Design D. - -### Constraint Failures - -Timeout is decisive. `TimeoutLayer` wraps service calls and replaces the -executor in `OperationContext`. A method decorator cannot reproduce the -executor behavior used by concurrent block operations. +## Design C: Exact-Release Shared Native Runtime -Foyer is also stateful and intercepts operation bodies. A language decorator -can implement a different cache, but it cannot claim native `FoyerLayer` -equivalence without a much larger protocol. - -### Appropriate Use - -Use this design when service extensibility matters but native layer parity does -not. It does not meet this proposal's layer requirement. - -## Design C: Exact-Build Shared Native Runtime - -### Seam +### Shared Runtime Seam The binding adapter and all native packages use one shared native runtime -module. That module owns OpenDAL/Tokio types and registries. Extensions register -erased service or layer factories compiled for exactly that runtime build. -Node.js can place an environment adapter over those process-scoped resources; -it does not create a second OpenDAL graph for each Worker. +module. That module owns OpenDAL and Tokio types and registries. Each extension +registers one erased service or layer factory compiled for exactly the same +OpenDAL release. Node.js can place an environment adapter over those +process-scoped resources; it does not create a second OpenDAL graph for each +Worker. ```text language adapter -> extension runtime <- service/layer native libraries @@ -156,10 +107,11 @@ language adapter -> extension runtime <- service/layer native libraries runtime-owned Operator ``` -The loader first reads a stable, C-layout bootstrap descriptor. It validates an -exact runtime build ID before entering any build-specific internal interface. +The loader reads a bounded JSON bootstrap document through a small C calling +convention. It validates the exact OpenDAL version and target before entering a +release-specific internal interface. -### Strengths +### Shared Runtime Strengths - Preserves the native `Operator`, `Layer`, and `OperationContext` model. - Keeps one OpenDAL core, Tokio runtime, and shared HTTP/context graph. @@ -168,98 +120,30 @@ exact runtime build ID before entering any build-specific internal interface. - Isolates native dependencies through lazy activation. - Shares the extension SDK and conformance suite across language bindings. -### Limitations +### Shared Runtime Limitations - Rust ABI stability is not promised. Every native package rebuilds for each - runtime build ID. + OpenDAL release. - Cross-platform shared-library discovery, repair, rpath/install-name behavior, Windows DLL lookup, and symbol visibility require substantial engineering. -- Package managers must prevent or clearly reject mixed runtime release trains. -- One physical extension artifact shared across language ecosystems requires proof; - the initial design only requires shared source and contracts. +- Package managers must prevent or clearly reject mixed release trains. +- One physical extension artifact shared across language ecosystems requires + proof; the initial design requires only shared source and contracts. - Native packages remain trusted code and cannot be unloaded safely in the initial design. -### Appropriate Use +### Shared Runtime Use This is the recommended balance when install-time composition and complete -native layers matter, but old extension binaries do not need to survive runtime +native layers matter, but old extension binaries do not need to survive OpenDAL upgrades. -## Design D: Stable C Operation Interface - -### Seam - -The host and each native extension exchange only C-layout function tables, opaque -reference-counted handles, fixed-width scalars, buffers, errors, and versioned -wire values. Rust values never cross the seam. - -```text -language adapter -> host operation graph -> stable C ABI -> extension operation graph -``` - -To support arbitrary services and layers, the interface must represent: - -- Service construction, capabilities, and every operation. -- Readers, writers, listers, deleters, copiers, and streams. -- Futures, polling or completion, waking, and cancellation. -- Buffer ownership and creator-side destruction. -- Errors and extensible operation options. -- HTTP and executor resources in `OperationContext`. -- Separate service and context hooks for layers. - -### Strengths - -- Extensions can use different OpenDAL core and Rust compiler versions. -- A correctly versioned extension binary can survive host upgrades. -- Native extensions can be shared by Python, Ruby, and Node.js on one target. -- Other C-ABI languages can implement extensions. - -### Limitations - -- The C interface becomes a second stabilized representation of OpenDAL's raw - operation model. -- Async cancellation, stream ownership, context composition, and wire evolution - create a large unsafe verification surface. -- Old extensions may be binary compatible while lacking new operation semantics. - They must reject required fields they do not understand rather than silently - dropping them. -- Each extension may embed its own OpenDAL/Tokio graph, increasing size and - making cross-extension layers more expensive. - -### ABI Evolution - -If OpenDAL selects this design, it must govern the operation ABI independently -from every language binding: - -- Every function table starts with a structure size and ABI version. -- A minor ABI revision can append optional methods but cannot reorder fields or - change existing ownership or semantics. -- A breaking layout, ownership rule, or required semantic change increments the - ABI major. -- Extensible wire values distinguish required and optional fields. An old - extension rejects an unknown required field instead of silently losing it. -- Every output has a creator-provided destructor. -- Task polling and mutable operation bodies define concurrency and cancellation - explicitly. -- Loader negotiation checks ABI major, wire major, target identity, and required - interface IDs before it exchanges handles. - -Stable ABI compatibility does not mean that an old extension implements every -new operation. It means the host can load the extension safely, negotiate its -capabilities, and receive an explicit unsupported result. - -### Appropriate Use - -Choose this design only when the project commits to long-term binary extension -governance. A service-only C interface is not enough for the layer requirement. - ## Constraint Comparison ### S3 and WebDAV -All designs must keep URI interpretation inside the service implementation. -The shared host passes a raw URI and explicit options instead of maintaining a +Both designs keep URI interpretation inside the service implementation. The +shared host passes a raw URI and explicit options instead of maintaining a central configuration schema. This matters because generic precedence is insufficient: @@ -269,43 +153,33 @@ This matters because generic precedence is insufficient: - WebDAV derives `https://authority` as its endpoint, overwriting an endpoint option supplied earlier. -Designs B, C, and D can preserve these rules if the package owns construction. -A host-owned universal schema is rejected in every design. - ### HDFS Design A cannot guarantee that the base artifact loads without HDFS native -dependencies when HDFS is compiled into it. Designs B, C, and D can register a -lightweight manifest and activate HDFS only at construction. `hdfs` and -`hdfs-native` remain independent packages and schemes. +dependencies when HDFS is compiled into it. Design C registers a JSON manifest +and activates HDFS only at construction. `hdfs` and `hdfs-native` remain +independent packages and schemes. ### Foyer Foyer requires asynchronous cache creation and a reusable stateful handle. -Design C holds the native layer in the shared runtime. Design D represents it -through a reference-counted layer handle. Design B can provide a language-level -cache but cannot claim arbitrary native layer composition. +Design C holds the native layer in the shared runtime. -Foyer's current key uses path and optional version, not service identity. No -design should advertise one cache handle as isolated across unrelated storages +Foyer's current key uses path and optional version, not service identity. The +design must not advertise one cache handle as isolated across unrelated storages until the package adds namespacing or restricts the handle to one logical storage. ### Timeout and Throttle -Designs A, C, and D can preserve both Timeout service wrapping and executor -wrapping. Design B cannot do so through method delegation alone. - -One Throttle layer owns shared limiter state. Designs C and D preserve the -identity of the layer handle so applying it to two operators shares a quota. -The extension adapter must validate positive bandwidth and burst values before -calling a constructor that currently asserts them. +Both designs preserve Timeout service wrapping and executor wrapping. One +Throttle layer owns shared limiter state. Design C preserves the identity of the +layer handle so applying it to two operators shares a quota. The extension +adapter must validate positive bandwidth and burst values before calling a +constructor that currently asserts them. -Timeout must be inside Retry. Applying Timeout and then Retry gives each retry -attempt a deadline. Applying Retry and then an outer Timeout is cancellation -unsafe because Timeout can drop Retry before it restores operation-body state. -An extension design must preserve the safe order and must not advertise an -outer whole-sequence Timeout as equivalent. +Both designs must preserve the canonical Timeout/Retry ordering defined by the +[layer compatibility rules](compatibility.md#layer-compatibility-rules). ## Why Node.js Does Not Change the Decision @@ -317,17 +191,10 @@ Python and Ruby. Node.js already publishes target-specific native npm packages, which provides a useful packaging pattern for extension artifacts. It still needs an exact -runtime build check, lazy activation, per-Worker adapter state, and a clear rule -against transferring JavaScript wrappers between Node environments. +OpenDAL version check, lazy activation, per-Worker adapter state, and a clear +rule against transferring JavaScript wrappers between Node environments. ## Recommendation Prototype Design C with one language-neutral extension SDK and three binding -adapters. Preserve Design A as a supported custom-build path. Do not combine C -with B: a language-owned operator plus an exact-runtime capsule creates two -competing composition models and inherits both sets of lifetime rules. - -Reconsider Design D only if users require already-built third-party extensions to -survive OpenDAL runtime upgrades. That requirement should be explicit because -it changes the project from coordinating a release train to governing a stable -operation ABI. +adapters. Preserve Design A as a supported custom-build path. diff --git a/bindings/docs/dynamic-extensions/compatibility.md b/bindings/docs/dynamic-extensions/compatibility.md index 1cfd8618ade9..af3f8e427a68 100644 --- a/bindings/docs/dynamic-extensions/compatibility.md +++ b/bindings/docs/dynamic-extensions/compatibility.md @@ -17,73 +17,36 @@ # Extension Compatibility and ABI -Status: pre-RFC contract for the exact-build runtime candidate. OpenDAL does not -provide or guarantee this ABI today. +Status: pre-RFC contract for the exact-release runtime candidate. OpenDAL does +not provide or guarantee this interface today. -This document distinguishes public language compatibility, package discovery, -the safe loader bootstrap, and the build-specific native interface. A version -match in one category does not imply a match in another category. +The design uses one compatibility version: the OpenDAL version. A native +extension must target exactly the version used by the runtime. The target +identity remains a separate platform check, not another versioning scheme. The key constraint is deliberate: -> The candidate design provides a stable language interface and a stable -> loader handshake, but it does not provide a stable Rust extension ABI. +> The candidate design provides stable public language interfaces, but it does +> not provide a stable Rust extension ABI across OpenDAL releases. -Native extensions must match an exact runtime build ID. An extension rebuild is -required whenever that build ID changes. +Every native extension rebuilds for every OpenDAL release. Equal layouts in two +releases do not create a supported compatibility range. -## Compatibility Identities +## Public Language Interfaces -| Identity | Purpose | Stability rule | -| ------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------- | -| Binding interface version | Public Python, Ruby, or Node.js behavior | Governed independently by each binding's compatibility policy | -| Manifest revision | Text discovery and registration fields | Major for breaking syntax/meaning; minor additions must be ignorable | -| Extension SDK revision | Source interface used to build native extensions | Semantic versioning can describe source changes, but not native compatibility | -| Bootstrap ABI version | C-layout descriptor read before native activation | Stable major; minor revisions append bounded optional fields | -| Runtime build ID | Exact internal native interface and shared Rust types | Equality required; no compatibility range | -| Target identity | OS, architecture, ABI, libc/deployment floor, and target features | Must be compatible with the running process and runtime artifact | - -Equal OpenDAL versions, binding versions, Rust compiler versions, or SDK -revisions are individually insufficient to establish exact-runtime binary -compatibility. - -## Public Language Interface - -The public binding interface remains independent in each ecosystem: - -- Python versions `Operator`, `AsyncOperator`, exceptions, typing, and package - discovery according to the Python binding policy. -- Ruby versions `OpenDal::Operator`, layers, errors, and blocking behavior - according to the Ruby binding policy. -- Node.js versions JavaScript classes, promises, synchronous variants, streams, - and TypeScript declarations according to the Node.js binding policy. - -A public binding release may preserve all language-level behavior while -changing its runtime build ID and requiring every native extension to rebuild. -Conversely, an extension package can change a typed configuration helper -without changing the runtime ABI. - -Python `abi3` describes compatibility between a Python extension and CPython -versions. A [Python capsule](https://docs.python.org/3/c-api/capsule.html) can -carry an opaque bootstrap pointer through the stable CPython ABI. Neither fact -stabilizes Rust values behind that pointer. - -[Node-API](https://nodejs.org/api/n-api.html) describes compatibility between -the base addon and Node.js versions. Its own documentation explicitly excludes -external libraries from that guarantee. It does not make two napi-rs -`External` values from different addons interchangeable. - -The Ruby adapter remains subject to the Ruby and Magnus compatibility policy. -A language-neutral service/layer extension should not link Ruby and therefore -does not acquire a Ruby ABI dependency of its own. +Python, Ruby, and Node.js continue to version their public APIs according to +their existing binding policies. CPython `abi3`, Node-API, and Ruby or Magnus +compatibility describe the interface between a binding adapter and its language +runtime. They are informational for the native extension loader and do not +replace the exact OpenDAL version check. ## Configuration Value Contract -Factories receive language-neutral configuration values. Version 1 has this -closed grammar: +Factories receive language-neutral configuration values. The matching OpenDAL +release defines this closed grammar: ```text -ConfigValueV1 = +ConfigValue = Null | Bool(bool) | I64(i64) @@ -91,22 +54,22 @@ ConfigValueV1 = | F64(finite IEEE-754 binary64) | Utf8(string) | Bytes(byte sequence) - | List(sequence) - | Map(map) - | DurationNs(u64) + | List(sequence) + | Map(map) + | SignedDuration(seconds: i64, nanoseconds: i32) ``` -`DurationNs` is the only duration representation at the shared factory seam. -It covers zero through `u64::MAX` nanoseconds. Binding-specific interfaces can -accept seconds, milliseconds, or native duration objects, but their adapters -must define nanosecond rounding and reject negative, non-finite, or overflowing -values before producing `DurationNs`. Numeric conversion never silently -truncates, saturates, or wraps. +`SignedDuration` is the only duration representation at the shared factory +boundary. It matches `jiff::SignedDuration`: the nanosecond field has an +absolute value below one second and has the same sign as the seconds field when +both are non-zero. This preserves the full signed range without squeezing total +nanoseconds into one `i64`. -The grammar has its own major version. Adding a value variant, changing a -numeric interpretation, or changing container semantics creates a new major and -a new exact runtime build ID. A new runtime can implement more than one grammar -major explicitly, but it never reinterprets an old tag. +Binding-specific interfaces can accept seconds, milliseconds, or native +duration objects, but their adapters must define nanosecond rounding and reject +non-finite, non-canonical, or overflowing values. Each service or layer decides +whether its configuration accepts negative or zero durations. Numeric +conversion never silently truncates, saturates, or wraps. The decoder enforces a maximum nesting depth of 32, a maximum UTF-8 map key of 4 KiB, a maximum individual string or byte value of 16 MiB, a maximum of 65,536 @@ -114,237 +77,187 @@ entries in one list or map, and a maximum encoded request size of 64 MiB. It rejects invalid UTF-8, duplicate map keys, non-finite floats, unknown value tags, and values outside the declared numeric ranges before calling package code. -A package versions its configuration schema independently from this value -grammar. Schema behavior follows these rules: +A package owns its configuration fields, defaults, validation, credentials, and +redaction behavior. The matching OpenDAL release defines the factory request; +the runtime does not negotiate separate grammar or configuration-schema +versions. + +Schema behavior follows these rules: - A missing field selects its declared default or produces a missing-field error. - `Null` is valid only when the field is explicitly nullable. Optional and nullable are separate properties. -- A package rejects every field absent from its schema. Version 1 has no - implicit or client-declared "ignorable unknown" convention. -- A package rejects an unsupported schema revision. It can declare an explicit - set or range of revisions only when it implements each revision's defaults - and field meanings. +- A package rejects every field absent from its schema. - A schema marks credential and token fields as secret. The runtime never logs raw configuration values, and package errors identify a secret field without rendering its value. -`UriRequest` remains separate: it contains one UTF-8 URI and a map of UTF-8 +`UriRequest` remains separate. It contains one UTF-8 URI and a map of UTF-8 option names to UTF-8 option values, matching current iterator construction. The runtime applies the same size limits and never includes raw option values in loader diagnostics. Service code remains responsible for URI semantics and service-specific validation. -## Manifest Compatibility - -Language packages register a text manifest without activating native code. A -manifest contains at least: - -```text -manifest_revision -package_id -package_version -provided_services[] -provided_layers[] -native_artifact_path -native_entry_kind -native_entry_symbol -runtime_build_id -target_identity -manifest_digest +## JSON Manifest + +Each language package registers one JSON manifest without activating native +code. A manifest declares exactly one service or layer. A service can also +declare aliases for its canonical scheme. + +An illustrative service manifest is: + +```json +{ + "opendal_version": "0.55.0", + "package_id": "opendal-service-s3", + "package_version": "0.55.0", + "component": { + "kind": "service", + "id": "s3", + "aliases": [] + }, + "native_artifact_path": "lib/opendal_service_s3.so", + "native_entry_kind": "c-json", + "native_entry_symbol": "opendal_service_s3_bootstrap", + "target_identity": "x86_64-unknown-linux-gnu" +} ``` -The registry must validate sizes, encoding, IDs, duplicate names, and paths -before storing the manifest. It compares the declared runtime build ID and -target with the running artifact and rejects a mismatch without loading native -code. It must not store credentials, URI options, or service configuration in -registration metadata. - -Manifest evolution follows these rules: - -- A new major revision may add a required field or change field meaning. -- A new minor revision may add optional fields only. Unknown fields in a newer - minor are optional by definition and an older reader ignores them. -- Readers reject an unknown major revision. -- Writers do not omit a field required by their declared major revision. -- A package's embedded native descriptor must repeat the package identity, - runtime build ID, target identity, and manifest digest. Activation fails if - the text and native declarations differ. - -## Bootstrap ABI - -The bootstrap exists only to inspect and reject an extension safely before -entering the exact-build interface. It must not expose `Operator`, `Layer`, -trait objects, Rust strings, Rust enums, futures, Tokio handles, or allocator -ownership. - -An illustrative descriptor is: - -```c -typedef struct { - const unsigned char *data; - size_t len; -} opendal_bytes_v1; - -typedef struct { - uint64_t magic; - uint32_t struct_size; - uint16_t bootstrap_major; - uint16_t bootstrap_minor; - opendal_bytes_v1 package_id; - opendal_bytes_v1 manifest_digest; - opendal_bytes_v1 runtime_build_id; - opendal_bytes_v1 target_identity; - void (*exact_build_entry)(void); -} opendal_extension_header_v1; +The exact OpenDAL release defines the document fields and their meaning. The +loader rejects a manifest whose `opendal_version` differs from the runtime; it +does not negotiate manifest revisions. `package_version` remains package +metadata and does not establish native compatibility. + +The registry validates document size, UTF-8, JSON structure, IDs, aliases, and +artifact paths before storing the manifest. It must not store credentials, URI +options, or service configuration in registration metadata. + +## JSON Bootstrap + +The bootstrap exists only to reject an incompatible native artifact before the +runtime enters the release-specific interface. The package exports a +package-unique function with a fixed C calling convention. The function reports +the required document length, then writes UTF-8 JSON into a host-provided +bounded buffer. It does not transfer memory ownership across the boundary. + +The bootstrap document repeats these manifest fields: + +```json +{ + "opendal_version": "0.55.0", + "package_id": "opendal-service-s3", + "package_version": "0.55.0", + "component_kind": "service", + "component_id": "s3", + "target_identity": "x86_64-unknown-linux-gnu", + "entry_symbol": "opendal_service_s3_entry" +} ``` -The actual RFC must specify maximum lengths, pointer validity, descriptor -lifetime, encoding, alignment, and failure behavior. Descriptor memory should -be immutable and valid while the native library remains loaded. - -The bootstrap ABI follows these rules: +The RFC must define the maximum document length, pointer validity, encoding, +symbol lifetime, and failure behavior. The bootstrap follows these rules: - Every exported function uses an explicit C calling convention. - No panic or foreign exception crosses the call. -- Every structure begins with a size and version or is referenced by a - versioned descriptor. -- Minor versions append fields; they never reorder or reinterpret existing - fields. -- The loader checks `magic`, minimum size, major version, bounded byte slices, - target identity, manifest digest, and runtime build ID in that order. -- The loader does not call `exact_build_entry` after any failed check. +- The loader rejects a reported document length above the defined limit before + allocating or parsing it. +- The loader checks the OpenDAL version, package identity, component identity, + target identity, and entry symbol before entering package code. +- The JSON manifest and bootstrap document must identify the same package and + component. - The bootstrap reports incompatibility; it does not attempt ABI adaptation. -This stable bootstrap is intentionally too small to construct an operator. A -stable construction interface would be Design D from -[the alternatives](alternatives.md), not an incremental relaxation of the -exact-build rule. - -## Runtime Build ID - -The coordinated runtime build issues an opaque, collision-resistant build ID. -Extension authors must not type or copy that identity into source code. The -published SDK build tool validates the extension build inputs against the -runtime build manifest and generates both the embedded descriptor and the text -manifest. It fails the build if a required input cannot be verified. +The bootstrap does not expose `Operator`, `Layer`, trait objects, Rust strings, +Rust enums, futures, Tokio handles, or allocator ownership. -The build manifest behind the ID must cover all ABI-relevant inputs, -including: +## OpenDAL Release Compatibility -- Extension runtime and SDK layout revisions. -- Exact OpenDAL core source identity. -- Exact versions and features of dependencies whose types or globals cross the - seam. -- `rustc -vV`, target specification, target features, and ABI-relevant codegen - options. -- Panic strategy, allocator/ownership policy, and symbol/linkage model. -- The supported factory and handle layouts. +The runtime, extension SDK, and official extension packages form one coordinated +OpenDAL release. The SDK build tool pins the exact dependencies, compiler, +target, Cargo profile, Rust flags, panic strategy, and linkage policy used by +that release. It generates both the package manifest and embedded bootstrap +document instead of asking extension authors to copy compatibility metadata into +source code. Package-local dependencies such as S3 signing, XML parsing, `hdrs`, Foyer, or a -rate limiter do not need identical versions when their types and globals stay -inside the package. If one of their types crosses the seam, it becomes an -ABI-visible dependency and must enter the runtime build identity. - -The supported SDK is a coordinated build tool, not only a Rust crate. It pins -the exact SDK dependencies and lock state, checks the compiler and target, -normalizes the permitted Cargo profile and Rust flags, verifies the panic and -linkage policy, and records the actual inputs in a machine-readable build -report. Release CI compares that report with the generated descriptor before it -publishes an artifact. - -Build-ID equality is a necessary accidental-mismatch check, not attestation and -not proof that arbitrary native code is compatible or well behaved. Official -packages use the coordinated release pipeline. A supported third-party source -package must build through the published SDK toolchain and pass the same -conformance suite. An arbitrary binary that merely claims or forges an issued -ID is trusted, unsupported in-process code; the loader cannot make it safe. - -The project must never replace a published runtime artifact with different bits -or a different build ID under the same package version. A rebuild receives a -new immutable runtime release. +rate limiter can use different versions when their types and globals remain +inside the package. Any value exchanged through the internal interface follows +the exact SDK contract for that OpenDAL release. + +The project never replaces a published runtime artifact with different bits +under the same OpenDAL version. A changed artifact requires a new release. ## Change Impact -| Change | Binding interface version | Runtime build ID | Extension action | -| ------------------------------------------------------------- | ------------------------------------------ | ---------------- | -------------------------------------------------- | -| Install another compatible service/layer | Unchanged | Unchanged | Register the new manifest | -| Add an optional service config field | Service package policy | Unchanged | Republish only that package if native code changes | -| Change a public language method incompatibly | Binding policy requires a breaking release | Maybe | Rebuild only if native inputs change | -| Add an optional manifest field | Manifest minor | Unchanged | Old readers ignore it | -| Append a bootstrap descriptor field | Bootstrap minor | Unchanged | Old loaders use the recorded structure size | -| Change an exact SDK handle or factory layout | Unchanged or additive language release | New ID | Rebuild every native extension | -| Change ABI-visible OpenDAL core code or features | Usually unchanged at language level | New ID | Rebuild every native extension | -| Change `rustc`, target ABI, panic strategy, or linkage inputs | Unchanged | New ID | Rebuild every native extension | -| Change only a package-private dependency | Unchanged | Unchanged | Rebuild only that package | + + +| Change | OpenDAL version | Extension action | +| ------------------------------------------- | --------------- | ------------------------------ | +| Install another compatible component | Unchanged | Register the new manifest | +| Change package-private code or dependencies | Unchanged | Republish only that package | +| Change a public binding method only | Per binding | No native rebuild | +| Change the manifest or bootstrap contract | New release | Rebuild every native extension | +| Change an SDK handle or factory layout | New release | Rebuild every native extension | +| Change ABI-visible OpenDAL code or features | New release | Rebuild every native extension | +| Change compiler, panic, or linkage inputs | New release | Rebuild every native extension | -This table describes compatibility impact, not release numbering. Each binding -and package still follows its own published version policy. + -## Exact-Build Internal Interface +## Exact-Release Internal Interface -After the bootstrap succeeds, the runtime may use a build-specific factory +After bootstrap validation, the runtime enters a release-specific factory interface generated by the extension SDK. That interface can exchange -runtime-owned operator and layer handles or exact-build Rust adapters. +runtime-owned operator and layer handles or exact-release Rust adapters. The following constraints apply: -- The interface is compatible only with one runtime build ID. +- The interface is compatible with exactly one OpenDAL release. - The [Rust ABI has no stability guarantees](https://doc.rust-lang.org/reference/items/external-blocks.html). -- Extensions rebuild for every new runtime build ID, including patch releases when - the ABI-relevant build changes. -- Equal layouts observed in two builds do not create a supported compatibility - range. - The loader rejects a mismatch before registering factory pointers. -- Package-local code catches panics before returning across the seam. -- Ownership remains on the creating side unless an exact SDK handle explicitly +- Package-local code catches panics before returning across the boundary. +- Ownership remains on the creating side unless an SDK handle explicitly transfers it. - -The first implementation should prefer opaque runtime handles and generated -adapters even inside the exact-build interface. This reduces the number of Rust -types that can accidentally become ABI-visible and makes a future stable ABI -easier to evaluate. +- Opaque runtime handles are preferred over exposing Rust types directly. ## Target Identity -A runtime build ID does not make an artifact portable across targets. The +An equal OpenDAL version does not make an artifact portable across targets. The loader must also match: - Operating system and architecture. - Pointer width, endianness, and calling convention. - Linux libc family and minimum version where applicable. - macOS deployment target and architecture. -- Windows toolchain/runtime family. +- Windows toolchain and runtime family. - Required CPU target features. - Language adapter variant when the runtime is not physically shared. -A runtime release may issue different build IDs for Python, Ruby, and Node.js -even when all three use the same extension SDK source. Cross-language reuse of -one physical native artifact is supported only after artifact-level tests prove -that those adapters load the same runtime identity on that target. +Cross-language reuse of one physical native artifact is supported only after +artifact-level tests prove that the adapters load the same runtime identity on +that target. ## Loader State Machine ```text unregistered | - | register validated text manifest + | register validated JSON manifest v registered | | first construction v -loading -> bootstrap validated -> factories installed -> active - | | | - +---- error -----+----------------------+--> failed +loading -> bootstrap validated -> factory installed -> active + | | | + +---- error -----+--------------------+--> failed ``` -The loader must provide these properties: +The loader provides these properties: -1. Registration of all names in one manifest is atomic. +1. One manifest registers one component and its service aliases atomically. 2. An identical registration is idempotent. -3. A different owner for an existing name rejects the full registration. +3. A different owner for the component or an alias rejects registration. 4. Registry locks are released before filesystem access, `dlopen`, addon loading, or package code. 5. Only the requested package activates. @@ -355,44 +268,34 @@ The loader must provide these properties: 9. The initial implementation never unloads an activated native extension. The current core registry silently replaces an existing scheme. The extension -registry must not inherit that behavior. +registry must reject conflicts instead. ## Package Manager Constraints Package metadata provides an early compatibility diagnostic. The native loader -remains authoritative because package managers do not understand the embedded -runtime build ID. - -| Ecosystem | Proposed constraint | Additional requirement | -| --------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -| Python | Extension distribution requires an exact `opendal-runtime` release | Embedded build ID check; wheel target must match | -| Ruby | Extension gem requires an exact `opendal-runtime` release | Embedded build ID check; source/native gem policy remains explicit | -| Node.js | Extension uses an exact runtime peer dependency and target-specific optional packages | Embedded build ID check; reject a nested wrong `ProcessRuntime` | - -Extension package versions may evolve independently from their target runtime. -Each published version still declares exactly which runtime release it targets. -A service fix that does not change the runtime can publish a new service package -against the same runtime build ID. - -The compatibility aggregator named `opendal` installs the runtime and the -service/layer set provided by the current monolithic release. Minimal users -install the runtime distribution and selected extensions directly. Release -publication order is runtime, extensions, then aggregator. - -Dependency installation alone does not register an extension. Each aggregator -must guarantee registration using its binding's defined mechanism: mandatory -named entry-point resolution in Python, a generated registration-stub index in -Ruby, and explicit stub imports during Node.js module initialization. The -aggregator must register metadata only and preserve construction-time native +remains authoritative. + + + +| Ecosystem | Proposed constraint | Additional requirement | +| --------- | -------------------------------------------------------------------------------- | --------------------------------------------------------- | +| Python | Extension distribution requires the exact OpenDAL runtime release | Bootstrap version check; wheel target must match | +| Ruby | Extension gem requires the exact OpenDAL runtime release | Bootstrap version check; source/native policy is explicit | +| Node.js | Extension uses an exact runtime peer dependency and target-specific dependencies | Reject a nested incompatible `ProcessRuntime` | + + + +Dependency installation alone does not register an extension. Each binding uses +its defined discovery mechanism and preserves construction-time native activation. ## Service Compatibility Rules - A service factory receives the raw URI and explicit string options. - The service package owns `Configurator::from_uri`, aliases, validation, - credentials, and redaction. + credentials, redaction, and registry metadata. - Query options are merged before explicit options by `OperatorUri`, but the - service configurator may subsequently derive or replace fields from URI + service configurator can subsequently derive or replace fields from URI authority and path. - Capabilities belong to the constructed operator. Registration metadata does not promise static capabilities. @@ -409,9 +312,6 @@ overwriting an endpoint option. HDFS activation must be lazy and isolated from - A layer factory may complete asynchronously. - A layer handle retains native state and its extension library lease. -- A layer descriptor carries its canonical ID and declared composition safety - constraints. An operator handle retains the ordered IDs applied through the - supported binding interface. - Applying a layer returns a new operator and does not mutate the source operator. - Later applications are outer layers. The runtime does not sort or deduplicate @@ -423,95 +323,61 @@ Timeout and Retry have one cancellation-safe order. Applying Timeout first and Retry second places Retry outside Timeout, so each retry attempt has its own timeout. Applying Retry first and Timeout second places Timeout outside Retry; the timeout can drop Retry's future before Retry restores operation-body state. -Official bindings must preserve caller order and reject this known unsafe outer -Timeout composition when the operator's layer metadata makes it visible. They -must document it as unsupported when an opaque third-party layer prevents that -check. The runtime must never silently reorder the layers. - -Foyer live cache state and Throttle token history are not automatically -serializable. A package can define a declarative reconstruction policy, but it -must distinguish a reconstructed resource from the original live state. - -## Lifetime and Concurrency - -- Operator and immutable layer handles may be shared according to the native - OpenDAL contract. -- Mutable readers, writers, listers, deleters, copiers, and task handles must - document and enforce their allowed concurrency. -- A language wrapper can be collected while a derived operator remains active; - the runtime handle must keep native state alive. -- Cancellation must drop or abort the represented operation according to its - documented semantics. It must not detach work merely to simplify FFI. -- Destruction runs through the creator's function or exact SDK adapter. -- No callback, destructor, or wake operation may target an unloaded library. -- Reuse after `fork` is unsupported for Tokio, JVM, connection-pool, and cache - state unless a package explicitly documents and tests it. - -Node.js adds one constraint: JavaScript and Node-API values belong to one -`napi_env`. A loaded runtime native module owns a process-scoped -`ProcessRuntime`, including registries, native handle identity, activation -state, and library leases. Each environment owns a separate -`EnvironmentAdapter` and cleanup hooks. The adapter must not move -environment-bound values between the main thread, Workers, or Electron -contexts. +Bindings must preserve caller order and reject this known unsafe outer Timeout +composition when their metadata makes it visible. ## Error Contract -The shared runtime must expose stable error categories for: +The runtime reports stable error categories: -- Manifest invalid or conflicting registration. -- Package not installed or not registered. -- Native artifact missing or unsupported on this target. -- Bootstrap symbol or descriptor invalid. -- Runtime build ID, target, or manifest mismatch. -- Native dependency load failure. -- Service/layer configuration invalid. -- Asynchronous layer initialization failure or cancellation. -- Package panic contained at the extension seam. -- Normal OpenDAL operation errors. +```text +NotInstalled +ManifestInvalid +Conflict +Incompatible +UnsupportedTarget +NativeLoadFailed +BootstrapInvalid +FactoryFailed +LayerInitializationFailed +``` -Bindings map those categories into language-native exception classes while -retaining package ID, scheme/layer ID, and construction operation. Diagnostics -must not include credentials or an unredacted configuration map. +Language adapters map these categories to native exception classes while +retaining package ID, component ID, and construction operation. Diagnostics do +not include credentials or an unredacted configuration map. ## Security Constraints -Native extensions are trusted in-process code. Neither the bootstrap nor an -exact build ID provides a sandbox, signature verification, provenance, process -isolation, or protection from a malicious extension. +Native extensions are trusted in-process code. Neither JSON validation nor an +exact OpenDAL version provides a sandbox, signature verification, provenance, +process isolation, or protection from a malicious extension. -Compatibility validation still protects the supported binding contract from -accidental mismatches. The loader and official SDK must: +The loader and official SDK must: - Validate all untrusted lengths and identifiers before use. - Catch panics at every exported native entry and runtime-invoked callback. - Keep package symbols local where supported. - Load only a path supplied by an installed, explicitly selected package. - Pin libraries while any callback, vtable, task, or object may reference them. -- Keep credentials out of manifests, fingerprints, and conflict errors. +- Keep credentials out of manifests and conflict errors. Package signing and registry provenance remain package-release concerns. They do not change the in-process trust model. -Official runtime and extension artifacts must use unwind-capable panic handling -at extension entries. An artifact built with `panic=abort` cannot contain a -panic at the seam and still satisfy containment; the SDK and release pipeline -therefore reject that panic strategy for this interface. The build ID records -the strategy so a mismatched artifact also fails compatibility validation. +Official artifacts use unwind-capable panic handling at extension entries. The +SDK and release pipeline reject `panic=abort` for this interface. ## Unsupported Guarantees -The exact-build design does not guarantee: +The exact-release design does not guarantee: -- Loading an extension built for a different runtime build ID. -- Compatibility based only on equal OpenDAL or binding semantic versions. +- Loading an extension built for a different OpenDAL version. - Native library unloading. - Safe use of arbitrary extension code or secret redaction by third parties. - Serialization of live caches, limiters, runtimes, JVMs, or connection pools. - One physical extension binary across Python, Ruby, and Node.js. - Static service capabilities before configuration. -- Runtime installation of a missing package. Discovery only observes packages - already installed by the application's package manager. +- Runtime installation of a missing package. - Backend correctness, authorization, durability, or resource limits beyond the existing OpenDAL threat model. @@ -520,10 +386,10 @@ The exact-build design does not guarantee: Before publishing a native extension interface, CI must verify: - 1,000 manifest registrations without eager native activation. -- Atomic multi-name conflicts and idempotent duplicate registration. -- Mismatch rejection before storing callbacks. +- Single-component conflicts, alias conflicts, and idempotent registration. +- Version and target mismatch rejection before storing callbacks. - Concurrent activation and process-lifetime library pinning. -- S3 and WebDAV URI/configuration semantics and secret-free errors. +- S3 and WebDAV URI and configuration semantics and secret-free errors. - HDFS load isolation from the base, S3, WebDAV, and `hdfs-native`. - Foyer asynchronous construction, cancellation, invalidation, and handle lifetime. diff --git a/bindings/docs/dynamic-extensions/nodejs.md b/bindings/docs/dynamic-extensions/nodejs.md index 91e39a4a1aed..81297bcb8d53 100644 --- a/bindings/docs/dynamic-extensions/nodejs.md +++ b/bindings/docs/dynamic-extensions/nodejs.md @@ -20,7 +20,7 @@ Status: pre-RFC binding-specific design proposal. The current Node.js binding remains one native addon with compile-time service and layer selection. -The exact-build extension design is useful for Node.js. Node-API makes the base +The exact-release extension design is useful for Node.js. Node-API makes the base addon portable across supported Node.js versions, but it does not stabilize the Rust interface between separately built OpenDAL native packages. @@ -56,32 +56,25 @@ napi-rs implementation types as the extension contract. The proposed family is: ```text -@opendal/runtime Node-API adapter and JavaScript interface +opendal Node-API adapter and JavaScript interface @opendal/runtime-linux-x64-gnu target-specific runtime addon @opendal/service-s3 S3 JavaScript stub and types @opendal/service-s3-linux-x64-gnu target-specific S3 native addon/library @opendal/layer-timeout Timeout stub and types @opendal/layer-foyer Foyer stub and types -opendal compatibility aggregator and re-export ``` Each root extension package declares: -- An exact peer dependency on the compatible `@opendal/runtime` release. +- An exact peer dependency on the compatible `opendal` release. - Target packages as optional dependencies with `os`, `cpu`, and `libc` metadata where available. - ESM and CommonJS entry points that use one registration implementation. -- A mandatory embedded runtime build-ID check. +- A mandatory embedded OpenDAL version check. - A clear error for optional dependencies omitted during installation. -The unscoped `opendal` package preserves the current import and installs the -runtime plus the service/layer set provided by the current release. It re-exports -the runtime classes and registers those packages' text manifests. It does not -contain a second native runtime. - -Minimal applications depend directly on `@opendal/runtime` and selected -extensions. Package names remain provisional pending npm namespace and release -prototypes. +Applications depend on `opendal` and their selected extensions. Package names +remain provisional pending npm namespace and release prototypes. One root package plus several target packages per extension creates a large publication matrix. A registry with 1,000 manifest records can be efficient; @@ -94,7 +87,7 @@ The native architecture gives process resources and Node environment state different owners: ```text -loaded @opendal/runtime native module +loaded opendal native module ProcessRuntime OpenDAL core, Tokio, registries, activation state, handle identity process-lifetime native library leases @@ -138,18 +131,18 @@ The first Node prototype should compare two packaging implementations: 1. The `ProcessRuntime` loads a language-neutral native library directly from the package manifest. 2. An environment-bound JavaScript activator loads a target-specific Node-API - descriptor addon in the calling `napi_env`. + bootstrap addon in the calling `napi_env`. -Both implementations use the same descriptor, runtime build ID, factories, and -conformance suite. A literal separately installed shared Rust `dylib` is not a -design assumption; npm/pnpm/Yarn layouts, rpaths, and Windows DLL discovery must -prove it first. +Both implementations use the same JSON bootstrap document, OpenDAL version, +factory contract, and conformance suite. A literal separately installed shared +Rust `dylib` is not a design assumption; npm/pnpm/Yarn layouts, rpaths, and +Windows DLL discovery must prove it first. Direct host loading is the preferred starting point because it naturally -preserves text registration, process-level activate-once behavior, and +preserves JSON registration, process-level activate-once behavior, and construction-time native activation. -The Node-API descriptor variant requires this explicit environment-bound +The Node-API bootstrap variant requires this explicit environment-bound activation sequence: 1. The JavaScript registration stub gives its `EnvironmentAdapter` a text @@ -159,12 +152,12 @@ activation sequence: 3. The callback resolves the package's target artifact and synchronously loads its `.node` addon in the same `napi_env`. ESM and CommonJS wrappers call one shared loader implementation. -4. The addon uses the normal Node-API initializer and returns a `napi_external` - containing the C-layout descriptor. It does not return a napi-rs class or an - `External` shared with the base. -5. The base validates the descriptor before giving its language-neutral exact - entry to the `ProcessRuntime`. The process runtime installs the factories - atomically and records an explicit process-lifetime library lease. +4. The addon uses the normal Node-API initializer and returns the bounded JSON + bootstrap document. It does not return a napi-rs class or an `External` + shared with the base. +5. The base validates the document before giving its release-specific entry to + the `ProcessRuntime`. The process runtime installs the factory and records an + explicit process-lifetime library lease. 6. The environment adapter releases its activation callback after success. The factory does not retain `napi_env`, `napi_value`, JavaScript references, or thread-safe functions. @@ -182,7 +175,7 @@ adapts completion into the calling environment. Explicit registration avoids bundler-dependent side effects: ```javascript -import { Operator } from "@opendal/runtime"; +import { Operator } from "opendal"; import { registerS3 } from "@opendal/service-s3"; registerS3(); @@ -193,25 +186,10 @@ const op = Operator.fromUri("s3://photos/archive", { ``` `registerS3()` is idempotent for the same package and `ProcessRuntime`. It -registers a text manifest without activating native code. Under the descriptor +registers a JSON manifest without activating native code. Under the bootstrap addon prototype it also installs one environment-local activation callback; under direct host loading the manifest's artifact path is sufficient. -The compatibility `opendal` aggregator can register its dependency set during -module initialization so existing construction remains concise: - -```javascript -import { Operator } from "opendal"; - -const op = new Operator("s3", { - bucket: "photos", - region: "us-east-1", -}); -``` - -The aggregator imports registration stubs, not every native target library. -Native activation still occurs on first construction. - A package may offer a documented side-effect registration subpath for convenience, but it must mark that subpath appropriately for bundlers. The explicit function remains the unambiguous interface. @@ -263,19 +241,18 @@ later. Layer packages validate JavaScript numbers before calling Rust. Timeout values use non-negative integer milliseconds and convert to the shared -[`DurationNs`](compatibility.md#configuration-value-contract) representation. -The adapter rejects values above `u64::MAX` nanoseconds and never truncates, -saturates, or wraps them. Throttle bandwidth and burst must be positive integers -in the supported `u32` range so invalid input cannot reach the core -constructor's assertions. Other options convert through the same shared -`ConfigValueV1` grammar; native factories never receive JavaScript objects. +[`SignedDuration`](compatibility.md#configuration-value-contract) +representation. +The adapter rejects values outside `SignedDuration`'s `i64`-seconds range and +never truncates, saturates, or wraps them. Throttle bandwidth and burst must be +positive integers in the supported `u32` range so invalid input cannot reach the +core constructor's assertions. Other options convert through the same shared +`ConfigValue` grammar; native factories never receive JavaScript objects. Applying one Throttle or Foyer handle to several operators preserves shared -native state. Later `.layer()` calls remain outer layers. Timeout must be -applied before Retry so each attempt has a deadline. Applying an outer Timeout -after Retry is unsupported because it can cancel Retry before operation-body -state is restored. The adapter preserves order and rejects this known unsafe -composition when both layer IDs are visible. +native state. Later `.layer()` calls remain outer layers. The adapter preserves +the [canonical Timeout/Retry order](compatibility.md#layer-compatibility-rules) +and rejects the known unsafe composition when both layer IDs are visible. ## Async Operations and Cancellation @@ -313,8 +290,8 @@ design and should not silently fall back to this interface. ## Version and Error Behavior Node-API version compatibility and OpenDAL runtime compatibility remain -separate. The generated package-version checks used by a native loader are not -a replacement for a mandatory embedded build-ID handshake. +separate. The generated package-version checks provide an early diagnostic; the +embedded exact OpenDAL version check remains authoritative. Extension lifecycle errors should be JavaScript `Error` subclasses or errors with stable codes: @@ -335,7 +312,7 @@ reason string. ## Multiple Runtime Versions npm can install nested copies of a package. An extension stub may therefore see -a different `@opendal/runtime` instance from the one that created an operator. +a different `opendal` instance from the one that created an operator. The design applies three defenses: @@ -358,10 +335,10 @@ their creating `EnvironmentAdapter`. 3. Make compiled services/layers use the internal extension factory model. 4. Publish runtime and target packages using the existing platform-loader experience. -5. Extract S3 and Timeout as tracer package families and register them from the - `opendal` compatibility aggregator. +5. Extract S3 and Timeout as tracer package families and remove them from the + base package after their packages are available. 6. Add Foyer async creation and HDFS lazy activation as design gates. -7. Test both direct native loading and Node-API descriptor addons before +7. Test both direct native loading and Node-API bootstrap addons before selecting the physical linking model. 8. Publish the third-party SDK only after Worker, ESM/CommonJS, target, and lifetime conformance passes. diff --git a/bindings/docs/dynamic-extensions/python.md b/bindings/docs/dynamic-extensions/python.md index a76416d56101..a910e73dbd0e 100644 --- a/bindings/docs/dynamic-extensions/python.md +++ b/bindings/docs/dynamic-extensions/python.md @@ -55,43 +55,39 @@ adopt a new operator abstraction. The proposed release family is: ```text -opendal-runtime owns the `opendal` import package and native runtime +opendal owns the `opendal` import package and native runtime opendal-service-s3 contributes the S3 manifest, native code, and typing opendal-service-hdfs contributes libhdfs-backed HDFS lazily opendal-layer-timeout contributes Timeout opendal-layer-foyer contributes Foyer -opendal compatibility aggregator with no native extension ``` Every native service/layer distribution requires an exact -`opendal-runtime` release and embeds its runtime build ID. The `opendal` -aggregator depends on the runtime and the service/layer set shipped by the -current monolithic `opendal` wheel. This preserves: +`opendal` release and embeds that OpenDAL version in its JSON bootstrap +document. The base wheel remains installable on its own: ```console python -m pip install opendal ``` -A minimal application instead installs only the runtime and selected packages: +An application installs the base runtime and selected packages: ```console python -m pip install \ - opendal-runtime \ + opendal \ opendal-service-s3 \ opendal-layer-timeout \ opendal-layer-foyer ``` -The package names are provisional. The split between runtime and aggregator is -not: allowing both distributions to install the same native `opendal` files -would create ambiguous ownership and upgrades. +The extension package names are provisional. ## Import Layout The intended typed import layout is: ```text -opendal regular package owned by opendal-runtime +opendal regular package owned by the base distribution opendal.services namespace subpackage opendal.services.s3 supplied by opendal-service-s3 opendal.services.hdfs supplied by opendal-service-hdfs @@ -131,7 +127,7 @@ allow the runtime to find an installed package without importing every package. The resolver follows these rules: 1. `import opendal` loads only the base adapter and runtime. -2. An explicit service/layer import registers its text manifest. +2. An explicit service/layer import registers its JSON manifest. 3. Construction of an unregistered scheme must look up the one entry point with the matching canonical name and load only that registration stub. It reports a conflict if more than one distribution claims the name. @@ -141,9 +137,7 @@ The resolver follows these rules: Explicit imports remain useful for deterministic startup and access to typed configuration classes. Entry-point discovery preserves the current concise URI -path for callers that only need strings. This lookup is mandatory: dependencies -installed by the compatibility `opendal` aggregator do not register themselves -merely because pip installed them. +path for callers that only need strings. ## Proposed Operator Interface @@ -177,10 +171,10 @@ op = AsyncOperator.from_config(config) The base `from_config` runtime path accepts a generic mapping or service recipe. Package-local generated `TypedDict` or dataclass definitions provide field checking without extending one base `ServiceConfig` union. The service package -owns structured serialization and schema revisioning. +owns structured serialization and validation for the matching OpenDAL release. The adapter converts mappings to the shared -[`ConfigValueV1`](compatibility.md#configuration-value-contract) grammar. It +[`ConfigValue`](compatibility.md#configuration-value-contract) grammar. It rejects unsupported Python objects, cyclic containers, oversized values, unknown fields, and numeric overflow before package construction. Package-local types can expose Python-native values, but no `PyObject` crosses the factory @@ -237,16 +231,15 @@ Throttle accepts only positive integer `bandwidth` and `burst` values in the supported `u32` range. Python validation must reject invalid values before the native constructor can assert. -Timeout values remain finite, non-negative seconds at the Python interface. The +Timeout values remain finite, positive seconds at the Python interface. The adapter uses the current `Duration::try_from_secs_f64` rule, which rounds to the -nearest nanosecond with ties to even, and then emits `DurationNs`. It rejects -values above `u64::MAX` nanoseconds instead of saturating them. +nearest nanosecond with ties to even, and then emits `SignedDuration`. It +rejects values outside `SignedDuration`'s `i64`-seconds range instead of +saturating them. -Later `.layer()` calls are outer layers. Timeout must be applied before Retry, -which puts Retry outside Timeout and gives each attempt a deadline. Applying -Timeout after Retry is unsupported because the outer Timeout can cancel Retry -before it restores operation-body state. The binding preserves order and -rejects that known unsafe composition when it can observe both layer IDs. +Later `.layer()` calls are outer layers. The binding preserves the +[canonical Timeout/Retry order](compatibility.md#layer-compatibility-rules) and +rejects the known unsafe composition when it can observe both layer IDs. ## Async Behavior @@ -311,7 +304,7 @@ which artifacts can actually be shared: - The extension still needs one artifact per supported native target and libc or deployment floor. - One extension wheel can cover several Python versions only if all of those base - wheels load the same runtime build ID and compatible shared library. + wheels use the same OpenDAL version and compatible shared library. - Wheel repair must retain the intended shared runtime relationship instead of copying private runtime libraries into every extension under conflicting names. @@ -319,12 +312,12 @@ which artifacts can actually be shared: does not imply free-threaded compatibility. The libhdfs-backed HDFS wheel may have a smaller platform allowlist or ship as a -source distribution. Installing `opendal-runtime`, S3, WebDAV, or +source distribution. Installing `opendal`, S3, WebDAV, or `hdfs-native` must not load HDFS code or require Java/Hadoop. ## Migration -1. Add the runtime, descriptor, and registry internally while services/layers +1. Add the runtime, JSON bootstrap, and registry internally while services/layers remain compiled into the base wheel. 2. Make built-in adapters use the same internal factory interface intended for external packages. @@ -332,8 +325,8 @@ source distribution. Installing `opendal-runtime`, S3, WebDAV, or packages and re-export existing names. 4. Extract S3 and Timeout as tracer distributions. Keep Memory in the runtime because OpenDAL core always provides it. -5. Convert `opendal` into the compatibility aggregator before removing bundled - features from its runtime dependency. +5. Remove extracted components from the base wheel after their packages are + available. 6. Generate package-local configuration types from the Rust service metadata. 7. Validate WebDAV, HDFS, Foyer, and Throttle before declaring the interface complete. diff --git a/bindings/docs/dynamic-extensions/ruby.md b/bindings/docs/dynamic-extensions/ruby.md index 905ec78bb853..e4b3b81e5706 100644 --- a/bindings/docs/dynamic-extensions/ruby.md +++ b/bindings/docs/dynamic-extensions/ruby.md @@ -53,41 +53,34 @@ behavior. The proposed release family is: ```text -opendal-runtime owns `require "opendal"` and the shared runtime +opendal owns `require "opendal"` and the shared runtime opendal-service-s3 provides S3 registration and native artifacts opendal-service-hdfs provides libhdfs-backed HDFS lazily opendal-layer-timeout provides Timeout opendal-layer-foyer provides Foyer -opendal compatibility aggregator ``` -Each native service/layer gem requires an exact `opendal-runtime` release and -embeds its runtime build ID. The `opendal` compatibility gem depends on the -runtime and the extensions included by the current monolithic gem. This -preserves: +Each native service/layer gem requires an exact `opendal` release and embeds +that OpenDAL version in its JSON bootstrap document. The base gem remains +installable on its own: ```console gem install opendal ``` -Applications that need a smaller installation select the runtime and extensions -in their `Gemfile`: +Applications select the base runtime and extensions in their `Gemfile`: ```ruby -gem "opendal-runtime", "= " +gem "opendal", "= " gem "opendal-service-s3" gem "opendal-layer-timeout" gem "opendal-layer-foyer" ``` -The aggregator must not install files that conflict with the runtime gem. It is -metadata and dependency coordination, not a second owner of `lib/opendal.rb` or -the runtime native library. - ## Registration and Activation -Each extension gem contains a Ruby registration stub, a text manifest, and gem -metadata mapping its canonical service/layer IDs to that stub: +Each extension gem contains a Ruby registration stub, a JSON manifest, and gem +metadata mapping its canonical service or layer ID to that stub: ```ruby require "opendal/runtime" @@ -108,14 +101,9 @@ require "opendal/layers/foyer" Requiring an extension reads and registers metadata but does not activate its native library. The first service/layer construction performs native loading -and the exact build-ID check. - -The compatibility aggregator publishes a generated index of the registration -stubs provided by its exact dependencies. When `require "opendal"` activates -that aggregator, the runtime requires each indexed registration stub. Those -stubs register text manifests only; they do not load native libraries. +and the exact OpenDAL version check. -For a minimal installation, construction of an unregistered scheme must resolve +Construction of an unregistered scheme must resolve the one matching registration stub from installed gem metadata. The resolver reports duplicate claims, caches results and deterministic failures, handles aliases deterministically, and never requires every native extension at @@ -163,7 +151,7 @@ WebDAV, HDFS, and third-party gems retain their own configurator behavior, validation, credentials, and redaction. Typed Ruby configuration objects and hashes convert to the shared -[`ConfigValueV1`](compatibility.md#configuration-value-contract) grammar. The +[`ConfigValue`](compatibility.md#configuration-value-contract) grammar. The base adapter rejects symbols or objects without a declared conversion, cyclic containers, oversized values, unknown fields, and numeric overflow before calling package code. Native factories never retain Ruby objects. @@ -197,18 +185,16 @@ The current names remain compatibility adapters: Timeout values remain finite, non-negative seconds at the Ruby interface. The adapter uses the current `Duration::try_from_secs_f64` rule, which rounds to the -nearest nanosecond with ties to even, and then emits `DurationNs`. It rejects -values above `u64::MAX` nanoseconds instead of saturating them. +nearest nanosecond with ties to even, and then emits `SignedDuration`. It rejects +values outside `SignedDuration`'s `i64`-seconds range instead of saturating them. Throttle accepts only positive integer `bandwidth` and `burst` values in the supported `u32` range. Ruby validation must reject invalid values before the native constructor can assert. -Later layer calls are outer layers. Timeout must be applied before Retry, which -puts Retry outside Timeout and gives each attempt a deadline. Applying Timeout -after Retry is unsupported because the outer Timeout can cancel Retry before it -restores operation-body state. The binding preserves order and rejects that -known unsafe composition when it can observe both layer IDs. +Later layer calls are outer layers. The binding preserves the +[canonical Timeout/Retry order](compatibility.md#layer-compatibility-rules) and +rejects the known unsafe composition when it can observe both layer IDs. ## Asynchronous Layer Construction @@ -276,15 +262,15 @@ not include credentials or unredacted option hashes. platforms. - A language-neutral service/layer library should not link Ruby or Magnus. - Native extension gems still need artifacts for every supported OS, - architecture, libc/deployment floor, and runtime build ID. + architecture, libc/deployment floor, and OpenDAL version. - Source gems build against the exact SDK/runtime metadata and verify the - resulting embedded build ID. + resulting embedded OpenDAL version. - The runtime loads an explicit artifact path from the gem manifest and keeps the library pinned. - Linux symbol visibility, macOS install names, and Windows DLL discovery must be tested with gems installed in normal Bundler layouts. - A native-gem failure may fall back to a documented source build, but it must - not silently load a different runtime build. + not silently load a different OpenDAL release. The current native-gem matrix is best effort. Dynamic extensions should not claim broader binary coverage until runtime plus adapter artifacts pass an @@ -310,7 +296,8 @@ unless a package later defines explicit reinitialization behavior. proposed SDK shapes. 3. Add `via_iter`, `from_uri`, `layer`, `OpenDal::Layers`, and structured errors without removing current methods. -4. Extract S3 and Timeout as tracer gems and make `opendal` depend on them. +4. Extract S3 and Timeout as tracer gems and remove them from the base gem after + their packages are available. 5. Add gem-metadata resolution for compatibility constructors. 6. Validate WebDAV configurator behavior and HDFS lazy activation. 7. Validate Foyer initialization and Throttle sharing before publishing the From 230a6ec4de99598e58caa949cc90238f1d96bacd Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Sat, 1 Aug 2026 17:29:26 +0800 Subject: [PATCH 3/8] docs: retain dynamic extension alternatives --- bindings/docs/dynamic-extensions/README.md | 22 +- .../docs/dynamic-extensions/alternatives.md | 217 ++++++++++++++---- .../docs/dynamic-extensions/compatibility.md | 149 ++++++++++-- bindings/docs/dynamic-extensions/nodejs.md | 23 +- bindings/docs/dynamic-extensions/python.md | 6 +- bindings/docs/dynamic-extensions/ruby.md | 4 +- 6 files changed, 331 insertions(+), 90 deletions(-) diff --git a/bindings/docs/dynamic-extensions/README.md b/bindings/docs/dynamic-extensions/README.md index 57ed1ec0615e..4b2e3b8656bb 100644 --- a/bindings/docs/dynamic-extensions/README.md +++ b/bindings/docs/dynamic-extensions/README.md @@ -31,7 +31,7 @@ successful cross-platform packaging prototype and the OpenDAL RFC process. ## Documents -- [Design alternatives](alternatives.md) compares two possible extension +- [Design alternatives](alternatives.md) compares four possible extension seams and explains why Design C is the leading prototype candidate. - [Compatibility and ABI](compatibility.md) defines version and target checks, loader checks, lifetime rules, and supported stability guarantees. @@ -161,9 +161,9 @@ The adapter preserves each language's normal interface: streams, ESM/CommonJS exports, and Node-API. The project should share the native extension contract, not force one public -language API across all three bindings. The JSON manifest and bootstrap -documents, configuration value grammar, factory semantics, error categories, -and conformance suite are common. Constructor names, typing, duration syntax, +language API across all three bindings. The JSON manifest, bootstrap metadata, +configuration value grammar, factory semantics, error categories, and +conformance suite are common. Constructor names, typing, duration syntax, blocking behavior, and package discovery remain binding-specific. ### Extension runtime @@ -209,16 +209,18 @@ A native extension contains exactly one service or layer factory. The package exports one generated, package-unique bootstrap symbol and hides other package-local symbols where the platform permits it. -The bootstrap returns a bounded UTF-8 JSON document so the runtime can inspect -the package identity, target, and exact OpenDAL version through a small C -calling convention. Only after all checks pass may the runtime enter the -release-specific internal interface. That internal interface is not a stable -Rust ABI. +The bootstrap exposes the package identity, target, and exact OpenDAL version +through a small C calling convention. The [compatibility +contract](compatibility.md#bootstrap-encoding-alternatives) compares a bounded +JSON document with an exact-release C-layout descriptor instead of removing +either encoding before prototyping. Only after all checks pass may the runtime +enter the release-specific internal interface. That internal interface is not a +stable Rust ABI. Node.js packages are an exception to the unique-initializer rule when the native artifact is itself a Node-API addon: Node defines the addon initializer. The Node adapter must use a language-appropriate initializer and validate the -same package identity in the returned JSON document. +same package identity in the returned bootstrap metadata. ## Construction Requests diff --git a/bindings/docs/dynamic-extensions/alternatives.md b/bindings/docs/dynamic-extensions/alternatives.md index 6b9c05e2cc22..3f74ea4a625a 100644 --- a/bindings/docs/dynamic-extensions/alternatives.md +++ b/bindings/docs/dynamic-extensions/alternatives.md @@ -20,37 +20,43 @@ Status: pre-RFC design comparison. No alternative has been accepted or implemented. -This document compares the two implementation models that fit the project's -maintenance constraints. The designs differ in whether users select components -at build time or install them as packages later. +This document compares four ways to deliver OpenDAL services and layers to +language bindings. Designs B and D remain useful comparisons even though their +maintenance or semantic costs make them unsuitable for the current proposal. ## Decision Guide ```text Must packages compose after installation? no -> Design A: feature-selected monolithic binding - yes -> Design C: exact-release shared runtime + yes -> Must arbitrary native layers preserve core semantics? + no -> Design B: language-owned operator adapters + yes -> Must old binaries survive OpenDAL upgrades? + no -> Design C: exact-release shared runtime + yes -> Design D: stable C operation interface ``` OpenDAL should prototype Design C. Design A remains useful for custom builds. -The project does not plan to maintain language-specific operation adapters or a -second stable operation ABI. +Design B does not preserve arbitrary native layers and repeats an operation +adapter in every language. Design D preserves cross-version binaries only by +requiring OpenDAL to maintain a second operation model. ## Summary -| Property | A. Static build | C. Exact-release runtime | -| ------------------------------ | ----------------- | ------------------------ | -| Independently install services | No | Yes | -| Independently install layers | No | Yes | -| Full native layer semantics | Yes | Yes | -| One core/runtime graph | Yes | Yes | -| Third-party source extensions | Rebuild host | Yes, exact release | -| Cross-version native binaries | Not applicable | No | -| Native dependency isolation | No | Yes | -| Implementation cost | Lowest | High | -| Primary risk | Artifact variants | Packaging and linking | +| Property | A. Static build | B. Language adapters | C. Exact-release runtime | D. Stable C interface | +| ------------------------------ | ----------------- | -------------------- | ------------------------ | -------------------------- | +| Independently install services | No | Yes | Yes | Yes | +| Independently install layers | No | Language decorators | Yes | Yes | +| Full native layer semantics | Yes | No | Yes | Only represented semantics | +| One core/runtime graph | Yes | Usually no | Yes | Usually no | +| Third-party source extensions | Rebuild host | Yes | Yes, exact release | Yes | +| Cross-version native binaries | Not applicable | Language-dependent | No | Yes, within ABI rules | +| Cross-language native package | No | No | Possible, not promised | Yes, by design | +| Native dependency isolation | No | Yes | Yes | Yes | +| Implementation cost | Lowest | Moderate | High | Highest | +| Primary risk | Artifact variants | Semantic loss | Packaging and linking | ABI breadth and safety | @@ -89,6 +95,52 @@ language caller -> one native binding -> compiled services and layers Keep this path for custom source builds and hermetic distributions. It does not meet the independently installable package requirement. +## Design B: Language-Owned Operator Adapters + +### Language Adapter Seam + +Each service package owns an operator and exposes a Python, Ruby, or JavaScript +operation protocol. The base binding delegates every operation through that +language protocol. Layers decorate language objects. + +```text +language Operator facade -> language operation protocol -> package-owned backend + ^ + | + language decorator +``` + +### Language Adapter Strengths + +- Uses normal language package-loading mechanisms. +- Allows pure-language third-party services without a Rust ABI. +- Lets each native package own its core version and runtime because Rust values + do not cross the package boundary. +- Isolates HDFS dependencies in the HDFS package. + +### Language Adapter Limitations + +- Repeats or generates the complete operation adapter for every language. +- Adds language calls to streaming and operation-body paths unless the adapter + adds another native batching interface. +- Prevents a native OpenDAL layer from wrapping an operator owned by another + extension. +- Eventually requires readers, writers, listers, deleters, copiers, + cancellation, HTTP context, and executor context to reproduce native layers. + +Timeout is decisive. `TimeoutLayer` wraps service calls and replaces the +executor in `OperationContext`. A method decorator cannot reproduce the +executor behavior used by concurrent block operations. Foyer is stateful and +intercepts operation bodies; a language decorator can implement a different +cache but cannot claim native `FoyerLayer` equivalence without a larger +protocol. + +### Language Adapter Use + +Keep this design as the service-only comparison. It does not meet the proposal's +complete native-layer requirement, and OpenDAL does not plan to maintain a full +operation adapter separately for every language. + ## Design C: Exact-Release Shared Native Runtime ### Shared Runtime Seam @@ -107,8 +159,10 @@ language adapter -> extension runtime <- service/layer native libraries runtime-owned Operator ``` -The loader reads a bounded JSON bootstrap document through a small C calling -convention. It validates the exact OpenDAL version and target before entering a +The loader reads compatibility metadata through a small C calling convention. +The [compatibility contract](compatibility.md#bootstrap-encoding-alternatives) +compares a JSON document with a C-layout descriptor for that metadata. The +loader validates the exact OpenDAL version and target before entering a release-specific internal interface. ### Shared Runtime Strengths @@ -122,15 +176,15 @@ release-specific internal interface. ### Shared Runtime Limitations -- Rust ABI stability is not promised. Every native package rebuilds for each +- Does not promise Rust ABI stability. Every native package rebuilds for each OpenDAL release. -- Cross-platform shared-library discovery, repair, rpath/install-name behavior, - Windows DLL lookup, and symbol visibility require substantial engineering. -- Package managers must prevent or clearly reject mixed release trains. -- One physical extension artifact shared across language ecosystems requires - proof; the initial design requires only shared source and contracts. -- Native packages remain trusted code and cannot be unloaded safely in the - initial design. +- Requires cross-platform shared-library discovery, repair, + rpath/install-name behavior, Windows DLL lookup, and symbol visibility work. +- Requires package managers to prevent or clearly reject mixed release trains. +- Needs artifact-level proof before one physical extension artifact can be + shared across language ecosystems. +- Treats native packages as trusted code and does not unload them in the initial + design. ### Shared Runtime Use @@ -138,13 +192,71 @@ This is the recommended balance when install-time composition and complete native layers matter, but old extension binaries do not need to survive OpenDAL upgrades. +## Design D: Stable C Operation Interface + +### Stable C Interface Seam + +The base binding and each native extension exchange only C-layout function +tables, opaque reference-counted handles, fixed-width scalars, buffers, errors, +and versioned wire values. Rust values never cross the package boundary. + +```text +language adapter -> base operation graph + | + v + stable C ABI + | + v + extension operation graph +``` + +To support arbitrary services and layers, the interface must represent: + +- Service construction, capabilities, and every operation. +- Readers, writers, listers, deleters, copiers, and streams. +- Futures, polling or completion, waking, and cancellation. +- Buffer ownership and creator-side destruction. +- Errors and extensible operation options. +- HTTP and executor resources in `OperationContext`. +- Separate service and context hooks for layers. + +### Stable C Interface Strengths + +- Allows extensions to use different OpenDAL core and Rust compiler versions. +- Lets a correctly versioned extension binary survive base-binding upgrades. +- Can share native extensions across Python, Ruby, and Node.js on one target. +- Allows other C-ABI languages to implement extensions. + +### Stable C Interface Limitations + +- Creates a second stabilized representation of OpenDAL's raw operation model. +- Creates a large unsafe verification surface around async cancellation, stream + ownership, context composition, and wire evolution. +- Can load an old extension that lacks new operation semantics, requiring + explicit negotiation and unsupported results for every evolution. +- Allows each extension to embed its own OpenDAL and Tokio graph, increasing + size and making cross-extension layers more expensive. + +Long-term ABI governance would require size-tagged function tables, append-only +minor evolution, explicit breaking revisions, creator-provided destructors, +and documented task polling and cancellation semantics. Binary compatibility +would mean safe loading and explicit capability negotiation, not automatic +support for every new operation. + +### Stable C Interface Use + +Keep this design as the comparison for cross-version binary compatibility. +OpenDAL does not plan to implement it because its maintenance and verification +costs exceed the current requirement. A service-only C interface would not meet +the native-layer requirement. + ## Constraint Comparison ### S3 and WebDAV -Both designs keep URI interpretation inside the service implementation. The -shared host passes a raw URI and explicit options instead of maintaining a -central configuration schema. +All designs keep URI interpretation inside the service implementation. The base +binding passes a raw URI and explicit options instead of maintaining a central +configuration schema. This matters because generic precedence is insufficient: @@ -153,41 +265,48 @@ This matters because generic precedence is insufficient: - WebDAV derives `https://authority` as its endpoint, overwriting an endpoint option supplied earlier. +Designs B, C, and D preserve these rules when the package owns construction. A +base-owned universal schema is rejected in every design. + ### HDFS Design A cannot guarantee that the base artifact loads without HDFS native -dependencies when HDFS is compiled into it. Design C registers a JSON manifest -and activates HDFS only at construction. `hdfs` and `hdfs-native` remain -independent packages and schemes. +dependencies when HDFS is compiled into it. Designs B, C, and D can register a +lightweight manifest and activate HDFS only at construction. `hdfs` and +`hdfs-native` remain independent packages and schemes. ### Foyer Foyer requires asynchronous cache creation and a reusable stateful handle. -Design C holds the native layer in the shared runtime. +Design C holds the native layer in the shared runtime. Design D represents it +through a reference-counted layer handle. Design B can provide a language-level +cache but cannot claim arbitrary native layer composition. -Foyer's current key uses path and optional version, not service identity. The -design must not advertise one cache handle as isolated across unrelated storages +Foyer's current key uses path and optional version, not service identity. No +design should advertise one cache handle as isolated across unrelated storages until the package adds namespacing or restricts the handle to one logical storage. ### Timeout and Throttle -Both designs preserve Timeout service wrapping and executor wrapping. One -Throttle layer owns shared limiter state. Design C preserves the identity of the -layer handle so applying it to two operators shares a quota. The extension -adapter must validate positive bandwidth and burst values before calling a -constructor that currently asserts them. +Designs A, C, and D preserve both Timeout service wrapping and executor +wrapping. Design B cannot do so through method delegation alone. + +One Throttle layer owns shared limiter state. Designs C and D preserve the +identity of the layer handle so applying it to two operators shares a quota. +The extension adapter must validate positive bandwidth and burst values before +calling a constructor that currently asserts them. -Both designs must preserve the canonical Timeout/Retry ordering defined by the -[layer compatibility rules](compatibility.md#layer-compatibility-rules). +Every viable design must preserve the canonical Timeout/Retry ordering defined +by the [layer compatibility rules](compatibility.md#layer-compatibility-rules). ## Why Node.js Does Not Change the Decision [Node-API](https://nodejs.org/api/n-api.html) is ABI-stable across supported -Node.js versions, so it is a strong seam between JavaScript and the base addon. -It does not stabilize OpenDAL's internal Rust types or external libraries used -by the addon. Therefore Node.js benefits from Design C for the same reason as -Python and Ruby. +Node.js versions, so it is a strong boundary between JavaScript and the base +addon. It does not stabilize OpenDAL's internal Rust types or external libraries +used by the addon. Therefore Node.js benefits from Design C for the same reason +as Python and Ruby. Node.js already publishes target-specific native npm packages, which provides a useful packaging pattern for extension artifacts. It still needs an exact @@ -197,4 +316,6 @@ rule against transferring JavaScript wrappers between Node environments. ## Recommendation Prototype Design C with one language-neutral extension SDK and three binding -adapters. Preserve Design A as a supported custom-build path. +adapters. Preserve Design A as a supported custom-build path. Keep Designs B +and D documented as rejected alternatives so future proposals can evaluate +whether their requirements justify the maintenance or semantic costs. diff --git a/bindings/docs/dynamic-extensions/compatibility.md b/bindings/docs/dynamic-extensions/compatibility.md index af3f8e427a68..bd2185a877f4 100644 --- a/bindings/docs/dynamic-extensions/compatibility.md +++ b/bindings/docs/dynamic-extensions/compatibility.md @@ -118,8 +118,7 @@ An illustrative service manifest is: "aliases": [] }, "native_artifact_path": "lib/opendal_service_s3.so", - "native_entry_kind": "c-json", - "native_entry_symbol": "opendal_service_s3_bootstrap", + "native_entry_symbol": "opendal_service_s3_bootstrap_v1", "target_identity": "x86_64-unknown-linux-gnu" } ``` @@ -133,15 +132,88 @@ The registry validates document size, UTF-8, JSON structure, IDs, aliases, and artifact paths before storing the manifest. It must not store credentials, URI options, or service configuration in registration metadata. -## JSON Bootstrap +## Bootstrap Encoding Alternatives + +The installed discovery manifest remains JSON so the registry can inspect it +without loading native code. The native bootstrap repeats its compatibility +metadata after the library loads. The bootstrap encoding is still a design +choice: OpenDAL should compare a bounded JSON document with an exact-release +C-layout descriptor instead of discarding either option before prototyping. + +For directly loaded native libraries, both encodings use a package-unique +exported function with a fixed C calling convention. They exist only to reject +an incompatible native artifact before the runtime enters the release-specific +interface. + +Both encodings sit behind the same stable bootstrap envelope and function +signature: + +```c +enum { + OPENDAL_BOOTSTRAP_JSON = 1, + OPENDAL_BOOTSTRAP_C_LAYOUT_V1 = 2, +}; + +typedef struct { + uint32_t struct_size; + uint32_t payload_encoding; + const unsigned char *payload; + size_t payload_len; +} opendal_bootstrap_result_v1; + +typedef int32_t (*opendal_bootstrap_fn_v1)( + opendal_bootstrap_result_v1 *result +); +``` + +The host initializes `struct_size` to `sizeof(opendal_bootstrap_result_v1)` and +zeroes the other fields before calling the package-unique symbol. Every version +1 bootstrap symbol ends in `_bootstrap_v1` and uses this signature. A future +incompatible envelope uses a new symbol suffix, so the loader never guesses a +function signature from unvalidated package metadata. + +The function returns one of these status codes: + +- `0`: Success. The result contains one recognized, bounded payload. +- `1`: Invalid argument, including a null result pointer. +- `2`: Unsupported bootstrap envelope, including an undersized `struct_size`. +- `3`: The package could not provide bootstrap metadata. + +The loader treats every non-zero or unknown status as `BootstrapInvalid` and +does not read the payload fields. On success, the payload is immutable +package-owned memory that remains valid while the library is loaded. The loader +rejects a null payload, a zero or excessive length, or an unknown encoding +before decoding it. -The bootstrap exists only to reject an incompatible native artifact before the -runtime enters the release-specific interface. The package exports a -package-unique function with a fixed C calling convention. The function reports -the required document length, then writes UTF-8 JSON into a host-provided -bounded buffer. It does not transfer memory ownership across the boundary. +A Node-API addon initializer cannot use a package-unique C initializer. It +returns the same status, payload-encoding discriminant, and bounded payload +through Node-API values. The environment adapter applies the same validation +before passing metadata to the process runtime. -The bootstrap document repeats these manifest fields: +Prototype runtimes can accept both payload encodings through this envelope for +comparison. A published OpenDAL release selects one encoding for its supported +SDK and official packages. The common function signature remains the same, so a +stale manifest cannot make the loader call the bootstrap with the wrong ABI. + + + +| Property | JSON payload | C-layout payload | +| ------------------ | ------------------------------------------------- | ------------------------------------------------ | +| Bootstrap call | Common version 1 envelope | Common version 1 envelope | +| Installed manifest | JSON | JSON | +| Native metadata | UTF-8 names and values | Size-tagged structure with bounded byte slices | +| Payload ownership | Immutable package memory | Immutable package memory | +| Validation surface | Length, UTF-8, JSON, fields | Pointer, length, size, alignment, and fields | +| Human inspection | Direct | Requires a decoding tool | +| Node-API transport | String or byte buffer | Wrapper around the native structure | +| Evolution | Schema follows the exact OpenDAL release | New layout needs a new payload discriminant | +| Main risk | Parser complexity and non-canonical serialization | Unsafe pointer, length, and alignment validation | + + + +### JSON Payload + +The JSON payload is a bounded UTF-8 document. An illustrative document is: ```json { @@ -155,22 +227,67 @@ The bootstrap document repeats these manifest fields: } ``` -The RFC must define the maximum document length, pointer validity, encoding, -symbol lifetime, and failure behavior. The bootstrap follows these rules: +The loader validates the envelope length before parsing. The RFC must define +canonical encoding where bytewise comparison matters; field comparison must not +depend on JSON object order. + +### C-Layout Payload + +The C-layout payload points to this illustrative version 1 metadata structure: + +```c +typedef struct { + const unsigned char *data; + size_t len; +} opendal_bytes; + +typedef struct { + uint32_t struct_size; + opendal_bytes opendal_version; + opendal_bytes package_id; + opendal_bytes package_version; + opendal_bytes component_kind; + opendal_bytes component_id; + opendal_bytes target_identity; + opendal_bytes entry_symbol; +} opendal_c_metadata_v1; +``` + +The package contract requires every returned pointer to reference immutable +package-owned memory for the declared lifetime. The loader can reject null or +misaligned pointers and invalid structural bounds, but it cannot prove that an +arbitrary in-process pointer is mapped safely. After the checkable pointer and +alignment checks, the loader requires `payload_len` to cover the `struct_size` +field. It then requires `struct_size` to contain every version 1 field and not +exceed `payload_len` before reading any byte slice. Version 1 defines the +complete layout needed to read the OpenDAL version; changing that layout +requires a new payload encoding discriminant. The RFC must define maximum slice +lengths, encoding, and structure lifetime. + +### Shared Bootstrap Rules + +Whichever encoding the prototype selects, the bootstrap follows these rules: - Every exported function uses an explicit C calling convention. - No panic or foreign exception crosses the call. -- The loader rejects a reported document length above the defined limit before - allocating or parsing it. +- Every package-unique bootstrap symbol uses the common envelope signature. - The loader checks the OpenDAL version, package identity, component identity, - target identity, and entry symbol before entering package code. -- The JSON manifest and bootstrap document must identify the same package and + target identity, and entry symbol before invoking the release-specific entry + point or factory. +- The JSON manifest and native bootstrap must identify the same package and component. -- The bootstrap reports incompatibility; it does not attempt ABI adaptation. +- The loader reports incompatible metadata; it does not attempt ABI adaptation. The bootstrap does not expose `Operator`, `Layer`, trait objects, Rust strings, Rust enums, futures, Tokio handles, or allocator ownership. +JSON is the leading candidate because the discovery manifest and native +metadata can share parsing and scalar-value conventions, and the Node-API +adapter can transport it without native structure access. The C-layout payload +remains a candidate if the prototype demonstrates simpler or safer activation +on the supported native targets. The selection must follow cross-platform +loader tests, not document preference alone. + ## OpenDAL Release Compatibility The runtime, extension SDK, and official extension packages form one coordinated diff --git a/bindings/docs/dynamic-extensions/nodejs.md b/bindings/docs/dynamic-extensions/nodejs.md index 81297bcb8d53..e40e65b46780 100644 --- a/bindings/docs/dynamic-extensions/nodejs.md +++ b/bindings/docs/dynamic-extensions/nodejs.md @@ -133,10 +133,10 @@ The first Node prototype should compare two packaging implementations: 2. An environment-bound JavaScript activator loads a target-specific Node-API bootstrap addon in the calling `napi_env`. -Both implementations use the same JSON bootstrap document, OpenDAL version, -factory contract, and conformance suite. A literal separately installed shared -Rust `dylib` is not a design assumption; npm/pnpm/Yarn layouts, rpaths, and -Windows DLL discovery must prove it first. +Both implementations use the same bootstrap metadata and selected encoding, +OpenDAL version, factory contract, and conformance suite. A literal separately +installed shared Rust `dylib` is not a design assumption; npm/pnpm/Yarn layouts, +rpaths, and Windows DLL discovery must prove it first. Direct host loading is the preferred starting point because it naturally preserves JSON registration, process-level activate-once behavior, and @@ -145,19 +145,20 @@ construction-time native activation. The Node-API bootstrap variant requires this explicit environment-bound activation sequence: -1. The JavaScript registration stub gives its `EnvironmentAdapter` a text +1. The JavaScript registration stub gives its `EnvironmentAdapter` a JSON manifest and a local activation callback. It loads no native target package. 2. First construction asks the adapter, on its JavaScript thread, to invoke that callback. 3. The callback resolves the package's target artifact and synchronously loads its `.node` addon in the same `napi_env`. ESM and CommonJS wrappers call one shared loader implementation. -4. The addon uses the normal Node-API initializer and returns the bounded JSON - bootstrap document. It does not return a napi-rs class or an `External` - shared with the base. -5. The base validates the document before giving its release-specific entry to - the `ProcessRuntime`. The process runtime installs the factory and records an - explicit process-lifetime library lease. +4. The addon uses the normal Node-API initializer and returns the bootstrap + status, encoding discriminant, and payload. The payload is JSON bytes or a + `napi_external` pointing to C-layout metadata. It does not return a napi-rs + class or an `External` shared with the base. +5. The base validates the status, length, and selected encoding before giving + its release-specific entry to the `ProcessRuntime`. The process runtime + installs the factory and records an explicit process-lifetime library lease. 6. The environment adapter releases its activation callback after success. The factory does not retain `napi_env`, `napi_value`, JavaScript references, or thread-safe functions. diff --git a/bindings/docs/dynamic-extensions/python.md b/bindings/docs/dynamic-extensions/python.md index a910e73dbd0e..6188e520bb16 100644 --- a/bindings/docs/dynamic-extensions/python.md +++ b/bindings/docs/dynamic-extensions/python.md @@ -63,8 +63,8 @@ opendal-layer-foyer contributes Foyer ``` Every native service/layer distribution requires an exact -`opendal` release and embeds that OpenDAL version in its JSON bootstrap -document. The base wheel remains installable on its own: +`opendal` release and embeds that OpenDAL version in its bootstrap metadata. The +base wheel remains installable on its own: ```console python -m pip install opendal @@ -317,7 +317,7 @@ source distribution. Installing `opendal`, S3, WebDAV, or ## Migration -1. Add the runtime, JSON bootstrap, and registry internally while services/layers +1. Add the runtime, bootstrap interface, and registry internally while services/layers remain compiled into the base wheel. 2. Make built-in adapters use the same internal factory interface intended for external packages. diff --git a/bindings/docs/dynamic-extensions/ruby.md b/bindings/docs/dynamic-extensions/ruby.md index e4b3b81e5706..9f2a6f3baa60 100644 --- a/bindings/docs/dynamic-extensions/ruby.md +++ b/bindings/docs/dynamic-extensions/ruby.md @@ -61,8 +61,8 @@ opendal-layer-foyer provides Foyer ``` Each native service/layer gem requires an exact `opendal` release and embeds -that OpenDAL version in its JSON bootstrap document. The base gem remains -installable on its own: +that OpenDAL version in its bootstrap metadata. The base gem remains installable +on its own: ```console gem install opendal From 2597f92633c4e57f29eb657af588851f027b2d70 Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Sat, 1 Aug 2026 22:13:37 +0800 Subject: [PATCH 4/8] docs: select shared binding runtime package --- bindings/docs/dynamic-extensions/README.md | 62 +++++-- .../docs/dynamic-extensions/alternatives.md | 25 +++ .../docs/dynamic-extensions/compatibility.md | 153 ++++++++++++++---- bindings/docs/dynamic-extensions/nodejs.md | 17 +- bindings/docs/dynamic-extensions/python.md | 19 ++- bindings/docs/dynamic-extensions/ruby.md | 14 +- 6 files changed, 224 insertions(+), 66 deletions(-) diff --git a/bindings/docs/dynamic-extensions/README.md b/bindings/docs/dynamic-extensions/README.md index 4b2e3b8656bb..513c5246e4ed 100644 --- a/bindings/docs/dynamic-extensions/README.md +++ b/bindings/docs/dynamic-extensions/README.md @@ -24,10 +24,13 @@ This design allows a language binding to install services and layers as independent packages. For example, an application can install only S3, Timeout, and Foyer without rebuilding or replacing the base binding. -The leading prototype candidate uses an exact-release shared native runtime. -Python, Ruby, and Node.js use different language interfaces over the same -native extension model. Selecting that candidate remains conditional on a -successful cross-platform packaging prototype and the OpenDAL RFC process. +The leading prototype candidate uses a shared native runtime package with a +small common protocol. Python, Ruby, and Node.js use different language +interfaces over the same runtime and native extension model. The proposed +release process publishes the runtime with the language bindings as one +coordinated release family. +Selecting that candidate remains conditional on a successful cross-platform +packaging prototype and the OpenDAL RFC process. ## Documents @@ -118,10 +121,11 @@ Python adapter Ruby adapter Node-API adapter S3 / WebDAV / HDFS Timeout / Foyer / Throttle ``` -The diagram shows a logical architecture. The first implementation may package -a separate runtime artifact for each language ecosystem. Sharing the SDK, -manifest rules, and conformance suite is required; sharing one physical runtime -library across Python, Ruby, and Node.js installations is not. +The diagram shows a logical architecture. The selected distribution model puts +the implementation in a shared runtime package instead of embedding a private +copy in each main binding package. Ecosystem-specific packages may wrap +target-specific artifacts, but they must resolve the same runtime release and +runtime identity when loaded into one process. ### Why one common runtime graph is required @@ -141,9 +145,17 @@ language adapter cannot preserve arbitrary native layer semantics. The common runtime graph therefore owns composition machinery and extensions register release-specific factories into it. Service packages still own their configuration, URI interpretation, credentials, redaction, and package-local -dependencies. "Common" means one graph for extensions loaded by one binding -runtime; it does not require one physical runtime library to be shared across -Python, Ruby, and Node.js. +dependencies. "Common" means that all bindings in one process resolve one +runtime identity rather than loading binding-private runtime graphs. + +The protocol can remain small even though the runtime owns substantial native +state. Independently built packages need only a stable way to acquire a runtime +API, declare their required protocol level, inspect the runtime's supported +protocol range, and invoke runtime-owned factories. +The API implementation can expose a small bootstrap surface, such as an API +lookup and a construction or registration entry point, backed by an extensible +function table. The function signatures, table layouts, value grammar, handle +ownership, and error rules together form the protocol. ### Binding adapter @@ -274,16 +286,32 @@ applications are outer layers, matching OpenDAL core behavior. ## Package Families -Each ecosystem needs a runtime distribution and independent extension packages: +The selected release family has three package roles: ```text -runtime distribution minimal binding and native runtime -service/layer packages exact dependency on that OpenDAL release +shared runtime package native runtime, protocol, registries, and handles +main binding package Python, Ruby, or Node.js public API and adapter +service/layer package manifest, language API, and native implementation ``` -A minimal installation uses the runtime distribution plus selected extension -packages. Independently installable does not imply independently ABI-versioned: -every native package must be rebuilt for a new OpenDAL release. +The main binding package declares the `required_runtime_protocol` that its +adapter needs. The runtime package exposes `minimum_runtime_protocol` and +`runtime_protocol` so the adapter can verify that requirement before using the +runtime API. A service or layer package also depends on the runtime package, +while its native artifact follows the exact-release extension compatibility +rules. A minimal installation uses the runtime package and one main binding +package; applications then add selected extension packages. + +The proposed release process publishes these packages together. Coordinated +publication gives every binding and official extension a consistent runtime +implementation, build contract, and compatibility matrix. Keeping the +implementation in its own package also makes upgrades and dependency +diagnostics easier to manage than embedding equivalent native code independently +in every binding. + +Independently installable does not imply independently ABI-versioned: every +native extension package must be rebuilt for every OpenDAL release, even when +the internal interface appears unchanged. The 1,000-manifest requirement measures registry behavior, not a commitment to publish 1,000 official package families. Before a split release, each ecosystem diff --git a/bindings/docs/dynamic-extensions/alternatives.md b/bindings/docs/dynamic-extensions/alternatives.md index 3f74ea4a625a..80582ff476a3 100644 --- a/bindings/docs/dynamic-extensions/alternatives.md +++ b/bindings/docs/dynamic-extensions/alternatives.md @@ -41,6 +41,31 @@ Design B does not preserve arbitrary native layers and repeats an operation adapter in every language. Design D preserves cross-version binaries only by requiring OpenDAL to maintain a second operation model. +## Runtime Distribution Alternatives + +Design C still needs a distribution choice. The protocol and the package are +separate concepts: independently built components always need a protocol, even +if that protocol consists of only a small API lookup and construction surface. +The runtime package determines who ships and owns its implementation. + + + +| Model | Distribution | Strengths | Limitations | +| ----- | ------------ | --------- | ----------- | +| Binding-embedded runtime | Every main binding package carries a private runtime implementation | Simplest installation; no separate runtime dependency; binding maintainers control loading | Duplicates native artifacts; can create multiple runtime identities in one process; lets bindings drift in behavior and build settings | +| Host-provided runtime | The main binding loads a runtime and exposes its API to extension packages | Avoids a separately visible runtime package; extensions reuse the host instance | Couples discovery to each host language; makes extensions depend on host loading order and adapter-specific mechanisms; does not naturally coordinate multiple languages | +| Shared runtime package | Main bindings and extension packages depend on one separately versioned runtime package | Gives all languages one runtime identity and implementation; centralizes compatibility checks, fixes, and native resources; fits coordinated OpenDAL releases | Requires cross-ecosystem native artifact resolution; needs package-manager rules against incompatible duplicate runtimes; adds an explicit dependency | + + + +OpenDAL selects the shared runtime package for the prototype. The project +will publish the runtime and language bindings as a coordinated release family, +which strengthens consistency and simplifies release management. +Although each binding could carry the same implementation, a separate package +provides one place to version, test, diagnose, and update that implementation. +Target-specific wheels, gems, and npm packages may act as ecosystem delivery +wrappers, but they must represent the same runtime release and protocol. + ## Summary diff --git a/bindings/docs/dynamic-extensions/compatibility.md b/bindings/docs/dynamic-extensions/compatibility.md index bd2185a877f4..ad52313f4496 100644 --- a/bindings/docs/dynamic-extensions/compatibility.md +++ b/bindings/docs/dynamic-extensions/compatibility.md @@ -17,12 +17,13 @@ # Extension Compatibility and ABI -Status: pre-RFC contract for the exact-release runtime candidate. OpenDAL does -not provide or guarantee this interface today. +Status: pre-RFC contract for the shared runtime candidate. OpenDAL does not +provide or guarantee this interface today. -The design uses one compatibility version: the OpenDAL version. A native -extension must target exactly the version used by the runtime. The target -identity remains a separate platform check, not another versioning scheme. +The design uses two compatibility axes. A language binding declares the runtime +protocol level that it requires. A native extension must target exactly the +OpenDAL version used by the runtime. The target identity remains a separate +platform check, not another versioning scheme. The key constraint is deliberate: @@ -40,10 +41,93 @@ compatibility describe the interface between a binding adapter and its language runtime. They are informational for the native extension loader and do not replace the exact OpenDAL version check. +## Runtime Protocol Compatibility + +The binding-to-runtime boundary uses a common, language-neutral protocol. The +protocol is distinct from the runtime package version and from the +exact-release native extension interface. + +Each main binding package declares one `required_runtime_protocol`. The runtime +package reports an inclusive supported range: + +- `minimum_runtime_protocol` identifies the oldest protocol contract that the + runtime still supports. +- `runtime_protocol` identifies the newest protocol capability that the runtime + provides. + +A binding is compatible when: + +```text +minimum_runtime_protocol + <= required_runtime_protocol + <= runtime_protocol +``` + +For example, a binding that requires protocol 20 can use a runtime that supports +protocols 18 through 23. Package metadata provides an early diagnostic, but the +loaded runtime reports the authoritative values before returning an API table. + +The runtime reports the range and acquires the requested API through one +bootstrap function equivalent to: + +```c +typedef struct { + uint32_t struct_size; + uint32_t minimum_runtime_protocol; + uint32_t runtime_protocol; +} opendal_runtime_protocol_info_v1; + +typedef struct opendal_runtime_api opendal_runtime_api; + +int32_t opendal_runtime_get_api_v1( + uint32_t required_runtime_protocol, + opendal_runtime_protocol_info_v1 *protocol_info, + const opendal_runtime_api **api +); +``` + +The runtime fills `protocol_info` even when the requested level is incompatible +and returns a null `api` in that case. On success, it returns an API table that +conforms to `required_runtime_protocol`. The binding performs this call before +converting configuration or acquiring a runtime-owned handle. The binding uses +no capability introduced after its required level, and the runtime does not +send that interaction a value, callback, or handle kind introduced after that +level. A newer runtime therefore preserves the complete behavior of every +protocol at or above `minimum_runtime_protocol`. + +The binding's requested level is also the interaction ceiling for an extension. +The loader rejects an extension whose `required_runtime_protocol` is greater +than the binding's requested level, even when the runtime itself provides that +newer level. An extension can require an older level because the exact-release +runtime and extension enter the interaction at the binding's requested level. + +An incompatible change to this bootstrap function uses a new exported symbol, +such as `opendal_runtime_get_api_v2`; it does not add a separately negotiated +ABI-major field to the normal protocol check. + +The protocol should expose the smallest practical surface. The exported +bootstrap function acquires a size-tagged API table; a registration or +construction function in that table performs the main operation. Even this +two-function interaction has a protocol: its function signatures, table layout, +`ConfigValue` representation, status codes, handle ownership, and lifetime +rules are the compatibility contract. + +Adding an ordinary service configuration field does not raise the runtime +protocol level because each service package owns its schema. Adding a new +shared `ConfigValue` variant, handle kind, factory capability, or lifetime rule +does raise the protocol level. The binding that first uses that capability then +raises `required_runtime_protocol`. The protocol does not contain language- or +service-specific identifiers such as `opendal.python.s3`. + +OpenDAL distributes the protocol implementation in the shared runtime package +and publishes that package with the language bindings. Ecosystem-specific +delivery wrappers must resolve the same runtime identity when multiple bindings +load in one process. + ## Configuration Value Contract -Factories receive language-neutral configuration values. The matching OpenDAL -release defines this closed grammar: +Factories receive language-neutral configuration values. The runtime protocol +defines this closed grammar at the binding's requested protocol level: ```text ConfigValue = @@ -77,21 +161,21 @@ entries in one list or map, and a maximum encoded request size of 64 MiB. It rejects invalid UTF-8, duplicate map keys, non-finite floats, unknown value tags, and values outside the declared numeric ranges before calling package code. -A package owns its configuration fields, defaults, validation, credentials, and -redaction behavior. The matching OpenDAL release defines the factory request; -the runtime does not negotiate separate grammar or configuration-schema -versions. +A package owns its configuration format, fields, defaults, validation, +credentials, redaction behavior, and any schema version. `ConfigValue` defines +only the shared transport vocabulary; it does not impose one schema model on +all bindings or packages. + +Each package therefore decides: -Schema behavior follows these rules: +- Whether a missing field selects a default or produces an error. +- Whether `Null` differs from a missing field. +- Whether to reject, ignore, or preserve unknown fields. +- How to represent and evolve package-specific configuration versions. +- Which credential and token fields require redaction. -- A missing field selects its declared default or produces a missing-field - error. -- `Null` is valid only when the field is explicitly nullable. Optional and - nullable are separate properties. -- A package rejects every field absent from its schema. -- A schema marks credential and token fields as secret. The runtime never logs - raw configuration values, and package errors identify a secret field without - rendering its value. +The runtime enforces the transport limits above and never logs raw configuration +values. Package validation errors must not render secret values. `UriRequest` remains separate. It contains one UTF-8 URI and a map of UTF-8 option names to UTF-8 option values, matching current iterator construction. @@ -110,6 +194,7 @@ An illustrative service manifest is: ```json { "opendal_version": "0.55.0", + "required_runtime_protocol": 20, "package_id": "opendal-service-s3", "package_version": "0.55.0", "component": { @@ -123,10 +208,12 @@ An illustrative service manifest is: } ``` -The exact OpenDAL release defines the document fields and their meaning. The -loader rejects a manifest whose `opendal_version` differs from the runtime; it -does not negotiate manifest revisions. `package_version` remains package -metadata and does not establish native compatibility. +The runtime protocol defines the common document fields and their meaning. The +loader first verifies that `required_runtime_protocol` does not exceed the +binding's already validated requested level, then rejects a native extension +manifest whose `opendal_version` differs from the runtime. It does not negotiate +native extension compatibility from the protocol level. `package_version` +remains package metadata and does not establish native compatibility. The registry validates document size, UTF-8, JSON structure, IDs, aliases, and artifact paths before storing the manifest. It must not store credentials, URI @@ -218,6 +305,7 @@ The JSON payload is a bounded UTF-8 document. An illustrative document is: ```json { "opendal_version": "0.55.0", + "required_runtime_protocol": 20, "package_id": "opendal-service-s3", "package_version": "0.55.0", "component_kind": "service", @@ -243,6 +331,7 @@ typedef struct { typedef struct { uint32_t struct_size; + uint32_t required_runtime_protocol; opendal_bytes opendal_version; opendal_bytes package_id; opendal_bytes package_version; @@ -271,9 +360,9 @@ Whichever encoding the prototype selects, the bootstrap follows these rules: - Every exported function uses an explicit C calling convention. - No panic or foreign exception crosses the call. - Every package-unique bootstrap symbol uses the common envelope signature. -- The loader checks the OpenDAL version, package identity, component identity, - target identity, and entry symbol before invoking the release-specific entry - point or factory. +- The loader checks the required runtime protocol, OpenDAL version, package + identity, component identity, target identity, and entry symbol before + invoking the release-specific entry point or factory. - The JSON manifest and native bootstrap must identify the same package and component. - The loader reports incompatible metadata; it does not attempt ABI adaptation. @@ -394,11 +483,11 @@ remains authoritative. -| Ecosystem | Proposed constraint | Additional requirement | -| --------- | -------------------------------------------------------------------------------- | --------------------------------------------------------- | -| Python | Extension distribution requires the exact OpenDAL runtime release | Bootstrap version check; wheel target must match | -| Ruby | Extension gem requires the exact OpenDAL runtime release | Bootstrap version check; source/native policy is explicit | -| Node.js | Extension uses an exact runtime peer dependency and target-specific dependencies | Reject a nested incompatible `ProcessRuntime` | +| Ecosystem | Main binding constraint | Native extension constraint | Additional requirement | +| --------- | ---------------------------------------- | --------------------------------------- | --------------------------------------------------------- | +| Python | Required runtime protocol | Exact OpenDAL runtime release | Bootstrap version check; wheel target must match | +| Ruby | Required runtime protocol | Exact OpenDAL runtime release | Bootstrap version check; source/native policy is explicit | +| Node.js | Required runtime protocol | Exact runtime and target dependencies | Reject a nested incompatible `ProcessRuntime` | diff --git a/bindings/docs/dynamic-extensions/nodejs.md b/bindings/docs/dynamic-extensions/nodejs.md index e40e65b46780..3cde1b9ffef8 100644 --- a/bindings/docs/dynamic-extensions/nodejs.md +++ b/bindings/docs/dynamic-extensions/nodejs.md @@ -57,6 +57,7 @@ The proposed family is: ```text opendal Node-API adapter and JavaScript interface +@opendal/runtime runtime loader and target resolver @opendal/runtime-linux-x64-gnu target-specific runtime addon @opendal/service-s3 S3 JavaScript stub and types @opendal/service-s3-linux-x64-gnu target-specific S3 native addon/library @@ -66,15 +67,21 @@ opendal Node-API adapter and JavaScript interface Each root extension package declares: -- An exact peer dependency on the compatible `opendal` release. +- An exact peer dependency on the compatible `@opendal/runtime` release. +- A compatible peer dependency on the `opendal` adapter when it exposes a + JavaScript API through that package. - Target packages as optional dependencies with `os`, `cpu`, and `libc` metadata where available. - ESM and CommonJS entry points that use one registration implementation. - A mandatory embedded OpenDAL version check. - A clear error for optional dependencies omitted during installation. -Applications depend on `opendal` and their selected extensions. Package names -remain provisional pending npm namespace and release prototypes. +The `opendal` adapter declares its `required_runtime_protocol`. +`@opendal/runtime` exposes its minimum and current protocol levels for the +adapter to check. Applications depend on `opendal` and their selected +extensions; the package manager resolves `@opendal/runtime` and its target +package. Package names remain provisional pending npm namespace and release +prototypes. One root package plus several target packages per extension creates a large publication matrix. A registry with 1,000 manifest records can be efficient; @@ -317,8 +324,8 @@ a different `opendal` instance from the one that created an operator. The design applies three defenses: -1. Exact peer dependencies make the intended singleton visible to the package - manager. +1. Exact runtime peer dependencies make the intended singleton visible to the + package manager. 2. Registration records the specific `ProcessRuntime` identity. 3. A layer/operator wrapper verifies process-runtime identity before native handle use and throws `OPENDAL_EXTENSION_INCOMPATIBLE` on mismatch. diff --git a/bindings/docs/dynamic-extensions/python.md b/bindings/docs/dynamic-extensions/python.md index 6188e520bb16..b1f5260bdbda 100644 --- a/bindings/docs/dynamic-extensions/python.md +++ b/bindings/docs/dynamic-extensions/python.md @@ -55,22 +55,26 @@ adopt a new operator abstraction. The proposed release family is: ```text -opendal owns the `opendal` import package and native runtime +opendal-runtime provides the shared native runtime +opendal owns the `opendal` import package and Python adapter opendal-service-s3 contributes the S3 manifest, native code, and typing opendal-service-hdfs contributes libhdfs-backed HDFS lazily opendal-layer-timeout contributes Timeout opendal-layer-foyer contributes Foyer ``` -Every native service/layer distribution requires an exact -`opendal` release and embeds that OpenDAL version in its bootstrap metadata. The -base wheel remains installable on its own: +The `opendal` distribution declares its `required_runtime_protocol`. The +`opendal-runtime` distribution exposes its minimum and current protocol levels +for the binding to check. Every native service/layer distribution requires an +exact `opendal-runtime` release and embeds that OpenDAL version in its bootstrap +metadata. Installing the base wheel resolves the runtime dependency: ```console python -m pip install opendal ``` -An application installs the base runtime and selected packages: +An application installs the main binding and selected packages; the package +manager resolves `opendal-runtime`: ```console python -m pip install \ @@ -303,8 +307,9 @@ which artifacts can actually be shared: - A language-neutral service/layer library should not link CPython. - The extension still needs one artifact per supported native target and libc or deployment floor. -- One extension wheel can cover several Python versions only if all of those base - wheels use the same OpenDAL version and compatible shared library. +- One extension wheel can cover several Python versions only if every base wheel + requires a protocol in the runtime's supported range and resolves the same + exact shared runtime release. - Wheel repair must retain the intended shared runtime relationship instead of copying private runtime libraries into every extension under conflicting names. diff --git a/bindings/docs/dynamic-extensions/ruby.md b/bindings/docs/dynamic-extensions/ruby.md index 9f2a6f3baa60..8a43569c211b 100644 --- a/bindings/docs/dynamic-extensions/ruby.md +++ b/bindings/docs/dynamic-extensions/ruby.md @@ -53,22 +53,26 @@ behavior. The proposed release family is: ```text -opendal owns `require "opendal"` and the shared runtime +opendal-runtime provides the shared native runtime +opendal owns `require "opendal"` and the Ruby adapter opendal-service-s3 provides S3 registration and native artifacts opendal-service-hdfs provides libhdfs-backed HDFS lazily opendal-layer-timeout provides Timeout opendal-layer-foyer provides Foyer ``` -Each native service/layer gem requires an exact `opendal` release and embeds -that OpenDAL version in its bootstrap metadata. The base gem remains installable -on its own: +The `opendal` gem declares its `required_runtime_protocol`. The +`opendal-runtime` gem exposes its minimum and current protocol levels for the +binding to check. Each native service/layer gem requires an exact +`opendal-runtime` release and embeds that OpenDAL version in its bootstrap +metadata. Installing the base gem resolves the runtime dependency: ```console gem install opendal ``` -Applications select the base runtime and extensions in their `Gemfile`: +Applications select the main binding and extensions in their `Gemfile`; Bundler +resolves `opendal-runtime`: ```ruby gem "opendal", "= " From 5d1489fb275954e3dfab962091b6f7ea98ee72fe Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Sun, 2 Aug 2026 15:54:53 +0800 Subject: [PATCH 5/8] docs: define native extension symbol isolation --- bindings/docs/dynamic-extensions/README.md | 12 +- .../docs/dynamic-extensions/alternatives.md | 2 + .../docs/dynamic-extensions/compatibility.md | 10 +- .../prototype/.cargo/config.toml | 7 + .../dynamic-extensions/prototype/.gitignore | 6 + .../dynamic-extensions/prototype/Cargo.lock | 1253 +++++++++++++++++ .../dynamic-extensions/prototype/Cargo.toml | 21 + .../dynamic-extensions/prototype/README.md | 78 + .../prototype/audit-elf-exports.sh | 31 + .../prototype/host/Cargo.toml | 16 + .../prototype/host/src/main.rs | 71 + .../dynamic-extensions/prototype/run-linux.sh | 27 + .../prototype/s3-extension/Cargo.toml | 19 + .../prototype/s3-extension/build.rs | 20 + .../prototype/s3-extension/exports.map | 11 + .../prototype/s3-extension/src/lib.rs | 25 + .../prototype/timeout-extension/Cargo.toml | 19 + .../prototype/timeout-extension/build.rs | 20 + .../prototype/timeout-extension/exports.map | 11 + .../prototype/timeout-extension/src/lib.rs | 30 + .../dynamic-extensions/symbol-isolation.md | 176 +++ core/services/s3/Cargo.toml | 2 +- 22 files changed, 1861 insertions(+), 6 deletions(-) create mode 100644 bindings/docs/dynamic-extensions/prototype/.cargo/config.toml create mode 100644 bindings/docs/dynamic-extensions/prototype/.gitignore create mode 100644 bindings/docs/dynamic-extensions/prototype/Cargo.lock create mode 100644 bindings/docs/dynamic-extensions/prototype/Cargo.toml create mode 100644 bindings/docs/dynamic-extensions/prototype/README.md create mode 100755 bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh create mode 100644 bindings/docs/dynamic-extensions/prototype/host/Cargo.toml create mode 100644 bindings/docs/dynamic-extensions/prototype/host/src/main.rs create mode 100755 bindings/docs/dynamic-extensions/prototype/run-linux.sh create mode 100644 bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml create mode 100644 bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs create mode 100644 bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map create mode 100644 bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs create mode 100644 bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml create mode 100644 bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs create mode 100644 bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map create mode 100644 bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs create mode 100644 bindings/docs/dynamic-extensions/symbol-isolation.md diff --git a/bindings/docs/dynamic-extensions/README.md b/bindings/docs/dynamic-extensions/README.md index 513c5246e4ed..fd3a29feabe6 100644 --- a/bindings/docs/dynamic-extensions/README.md +++ b/bindings/docs/dynamic-extensions/README.md @@ -38,6 +38,8 @@ packaging prototype and the OpenDAL RFC process. seams and explains why Design C is the leading prototype candidate. - [Compatibility and ABI](compatibility.md) defines version and target checks, loader checks, lifetime rules, and supported stability guarantees. +- [Native symbol isolation](symbol-isolation.md) defines export allowlists, + dependency ownership, artifact monitoring, and co-loading tests. - [Python design](python.md) defines Python packages, discovery, typing, and migration. - [Ruby design](ruby.md) defines Ruby gems, blocking construction, and @@ -218,8 +220,10 @@ layer. This isolates HDFS and similar native dependencies. ### Native extension A native extension contains exactly one service or layer factory. The package -exports one generated, package-unique bootstrap symbol and hides other -package-local symbols where the platform permits it. +exports one generated, package-unique bootstrap symbol. The final artifact must +hide every other package-local symbol and pass the [native symbol isolation +contract](symbol-isolation.md). Language-runtime initializers remain explicit +allowlist exceptions. The bootstrap exposes the package identity, target, and exact OpenDAL version through a small C calling convention. The [compatibility @@ -349,6 +353,10 @@ The design is not ready for an RFC decision until prototypes demonstrate: 6. Atomic registration and lazy lookup with 1,000 synthetic packages. 7. Target packaging on supported Linux, macOS, and Windows variants. 8. Python, Ruby, and Node.js adapters passing the same native conformance suite. +9. Final-artifact export, import, and native-dependency reports matching + explicit platform allowlists. +10. Co-loading independently compiled extensions without symbol interposition, + including real operations that exercise runtime-owned facilities. If those gates pass, the next deliverable is a `0000_*.md` RFC. These pre-RFC documents remain supporting analysis rather than evidence that the candidate diff --git a/bindings/docs/dynamic-extensions/alternatives.md b/bindings/docs/dynamic-extensions/alternatives.md index 80582ff476a3..0eb549d00181 100644 --- a/bindings/docs/dynamic-extensions/alternatives.md +++ b/bindings/docs/dynamic-extensions/alternatives.md @@ -205,6 +205,8 @@ release-specific internal interface. OpenDAL release. - Requires cross-platform shared-library discovery, repair, rpath/install-name behavior, Windows DLL lookup, and symbol visibility work. +- Requires final-artifact export allowlists and co-loading tests so private + dependency symbols cannot resolve across the runtime and extensions. - Requires package managers to prevent or clearly reject mixed release trains. - Needs artifact-level proof before one physical extension artifact can be shared across language ecosystems. diff --git a/bindings/docs/dynamic-extensions/compatibility.md b/bindings/docs/dynamic-extensions/compatibility.md index ad52313f4496..83959d135667 100644 --- a/bindings/docs/dynamic-extensions/compatibility.md +++ b/bindings/docs/dynamic-extensions/compatibility.md @@ -388,8 +388,9 @@ source code. Package-local dependencies such as S3 signing, XML parsing, `hdrs`, Foyer, or a rate limiter can use different versions when their types and globals remain -inside the package. Any value exchanged through the internal interface follows -the exact SDK contract for that OpenDAL release. +inside the package and their dynamic symbols satisfy the [native symbol +isolation contract](symbol-isolation.md). Any value exchanged through the +internal interface follows the exact SDK contract for that OpenDAL release. The project never replaces a published runtime artifact with different bits under the same OpenDAL version. A changed artifact requires a new release. @@ -425,6 +426,8 @@ The following constraints apply: - Ownership remains on the creating side unless an SDK handle explicitly transfers it. - Opaque runtime handles are preferred over exposing Rust types directly. +- All cross-artifact calls enter through validated bootstrap or SDK functions; + exact-release matching does not permit ambient Rust symbol resolution. ## Target Identity @@ -562,7 +565,8 @@ The loader and official SDK must: - Validate all untrusted lengths and identifiers before use. - Catch panics at every exported native entry and runtime-invoked callback. -- Keep package symbols local where supported. +- Enforce the export allowlist defined by the [native symbol isolation + contract](symbol-isolation.md). - Load only a path supplied by an installed, explicitly selected package. - Pin libraries while any callback, vtable, task, or object may reference them. - Keep credentials out of manifests and conflict errors. diff --git a/bindings/docs/dynamic-extensions/prototype/.cargo/config.toml b/bindings/docs/dynamic-extensions/prototype/.cargo/config.toml new file mode 100644 index 000000000000..4feecfab2d43 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/.cargo/config.toml @@ -0,0 +1,7 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +[target.'cfg(target_os = "linux")'] +rustflags = ["-C", "link-arg=-Wl,--exclude-libs,ALL"] diff --git a/bindings/docs/dynamic-extensions/prototype/.gitignore b/bindings/docs/dynamic-extensions/prototype/.gitignore new file mode 100644 index 000000000000..6274ef8dae7f --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/.gitignore @@ -0,0 +1,6 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +/target/ diff --git a/bindings/docs/dynamic-extensions/prototype/Cargo.lock b/bindings/docs/dynamic-extensions/prototype/Cargo.lock new file mode 100644 index 000000000000..bdab43529aa4 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/Cargo.lock @@ -0,0 +1,1253 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b25655df2c3cdd83c5e5b293b88acd880332b2ddadd7c30ac43144fdc0033da9" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cmov" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc-fast" +version = "1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5" +dependencies = [ + "digest 0.10.7", + "spin", +] + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "crypto-common 0.1.7", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer", + "const-oid", + "crypto-common 0.2.2", + "ctutils", +] + +[[package]] +name = "displaydoc" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "dlv-list" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f" +dependencies = [ + "const-random", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a88cf1f829d945f548cf8fec32c61b1f202b6d93b45848602fc02af4b12ad218" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-executor" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6754879cc9f2c66f88c6e5c35344bb0bdb0708b0352b1201815667c7eabc7458" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.3", +] + +[[package]] +name = "host" +version = "0.0.0" +dependencies = [ + "libloading", + "opendal-core", +] + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "hybrid-array" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" +dependencies = [ + "typenum", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +dependencies = [ + "defmt", +] + +[[package]] +name = "jiff-static" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "mea" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2640d335e7273dacdcf51044026139b2e269c3bb0dfc3f8cb3496b85e3f6a42c" +dependencies = [ + "slab", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opendal-core" +version = "0.58.1" +dependencies = [ + "anyhow", + "base64 0.23.0", + "bytes", + "futures", + "http", + "jiff", + "log", + "md-5", + "mea", + "percent-encoding", + "quick-xml", + "reqsign-core", + "serde", + "serde_json", + "tokio", + "url", + "uuid", + "web-time", +] + +[[package]] +name = "opendal-layer-timeout" +version = "0.58.1" +dependencies = [ + "opendal-core", + "tokio", +] + +[[package]] +name = "opendal-service-s3" +version = "0.58.1" +dependencies = [ + "base64 0.23.0", + "bytes", + "crc-fast", + "http", + "log", + "md-5", + "opendal-core", + "quick-xml", + "reqsign-aws-v4", + "reqsign-core", + "reqsign-file-read-tokio", + "serde", + "url", +] + +[[package]] +name = "ordered-multimap" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79" +dependencies = [ + "dlv-list", + "hashbrown", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "reqsign-aws-v4" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc883bc56889f3e4a419265c87facea222a921debc5c6f15c7fd8b68ec4b36b2" +dependencies = [ + "anyhow", + "bytes", + "form_urlencoded", + "hex", + "http", + "log", + "percent-encoding", + "quick-xml", + "reqsign-core", + "rust-ini", + "serde", + "serde_json", + "serde_urlencoded", + "sha1", +] + +[[package]] +name = "reqsign-core" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e38b44697c60a823705ccef85cb04d8e0527c9d16ed7c58bf1c6395bdd24ceb" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bytes", + "futures", + "hex", + "hmac", + "http", + "jiff", + "log", + "percent-encoding", + "sha1", + "sha2", + "windows-sys", +] + +[[package]] +name = "reqsign-file-read-tokio" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "688ff0ae421b8d4b92b53fdafaf53df2de28f428a9962edcf21702990b26f74b" +dependencies = [ + "anyhow", + "reqsign-core", + "tokio", +] + +[[package]] +name = "rust-ini" +version = "0.21.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7" +dependencies = [ + "cfg-if", + "ordered-multimap", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "s3-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-service-s3", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "timeout-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-layer-timeout", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "pin-project-lite", + "tokio-macros", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/bindings/docs/dynamic-extensions/prototype/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/Cargo.toml new file mode 100644 index 000000000000..be712fd7dcaa --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/Cargo.toml @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +[workspace] +members = ["host", "s3-extension", "timeout-extension"] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" + +[workspace.dependencies] +libloading = "0.8.9" +opendal-core = { path = "../../../../core/core", default-features = false } +opendal-layer-timeout = { path = "../../../../core/layers/timeout" } +opendal-service-s3 = { path = "../../../../core/services/s3" } diff --git a/bindings/docs/dynamic-extensions/prototype/README.md b/bindings/docs/dynamic-extensions/prototype/README.md new file mode 100644 index 000000000000..96b6b76c886b --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/README.md @@ -0,0 +1,78 @@ + + +# Exact-Release Dynamic Loading Prototype + +This prototype records one experiment behind the dynamic extension design. It +builds three final artifacts: + +- A host executable with its own statically linked `opendal-core`. +- An S3 `cdylib` with its own `opendal-core` and service dependencies. +- A Timeout `cdylib` with its own `opendal-core` and Tokio dependency. + +The runner builds each artifact in a separate Cargo invocation and target +directory, so Cargo does not unify their `opendal-core` feature graphs. The host +then loads both extensions with local visibility. The S3 extension creates an +`Operator`, the Timeout extension applies the real `TimeoutLayer`, and the host +reads the resulting operator information. All three artifacts compile against +the same source checkout, lockfile, compiler, target, profile, and Rust flags. + +On Linux, run: + +```console +./run-linux.sh +``` + +The script also audits the final ELF dynamic symbol tables. Each extension must +export only its package-unique bootstrap symbol. The local Cargo configuration +passes `--exclude-libs,ALL` to prevent symbols from statically linked dependency +archives from becoming dynamic exports. Explicit version scripts document the +intended allowlists, while the audit remains authoritative because Rust's +`cdylib` link can add exports after a user-supplied version script. + +## What It Proves + +The prototype shows that an exact-release internal adapter can move one current +`Operator` through separately linked S3 and Timeout artifacts on the tested +Linux toolchain. It also provides concrete artifacts for developing the [symbol +isolation contract](../symbol-isolation.md). It does not implement or conform +to the proposed SDK interface. + +## What It Does Not Prove + +This is design evidence, not a supported ABI or production loader: + +- The prototype exchanges a Rust value through an opaque pointer. The Rust ABI + remains unstable, so any change in compiler, target, profile, flags, features, + or dependency graph invalidates the experiment. +- The host ultimately destroys an allocation created and transformed by other + artifacts. This intentionally violates the proposed ownership contract. The + production SDK must use opaque handles and creator-side destructors rather + than treating this experiment as an ownership model. +- The prototype applies `TimeoutLayer` but does not execute an operation. + Therefore it does not prove that independently linked Tokio copies observe + compatible runtime state. +- The prototype does not provide the proposed shared runtime package, runtime + protocol negotiation, manifest validation, error contract, or library lease + implementation. +- The prototype covers ELF/Linux only. The design still requires equivalent + Mach-O and PE export enforcement and co-loading tests. +- The bootstrap functions return null on failure and omit structured errors. + The production contract must use the validated bootstrap envelope. + +The prototype should evolve into a conformance fixture only after the RFC +selects the exact-release interface representation. diff --git a/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh b/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh new file mode 100755 index 000000000000..94fda6c932a4 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +set -eu + +if [ "$#" -ne 2 ]; then + echo "usage: $0 ARTIFACT EXPECTED_SYMBOL" >&2 + exit 2 +fi + +artifact=$1 +expected=$2 +actual=$( + nm --dynamic --defined-only --extern-only --format=posix "$artifact" \ + | awk '{ print $1 }' \ + | sed 's/@.*//' \ + | LC_ALL=C sort -u +) + +if [ "$actual" != "$expected" ]; then + echo "unexpected exports in $artifact" >&2 + echo "expected: $expected" >&2 + echo "actual:" >&2 + echo "$actual" >&2 + exit 1 +fi + +echo "$artifact exports only $expected" diff --git a/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml new file mode 100644 index 000000000000..a6e1f425291c --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +[package] +name = "host" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[dependencies] +libloading.workspace = true +opendal-core.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/host/src/main.rs b/bindings/docs/dynamic-extensions/prototype/host/src/main.rs new file mode 100644 index 000000000000..6fb57c6eb087 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/host/src/main.rs @@ -0,0 +1,71 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0. + +use std::env; +use std::ffi::c_void; +use std::process::ExitCode; + +use libloading::{Library, Symbol}; +use opendal_core::Operator; + +type CreateOperator = unsafe extern "C" fn() -> *mut c_void; +type ApplyLayer = unsafe extern "C" fn(*mut c_void) -> *mut c_void; + +fn main() -> ExitCode { + match unsafe { run() } { + Ok(()) => ExitCode::SUCCESS, + Err(message) => { + eprintln!("{message}"); + ExitCode::FAILURE + } + } +} + +unsafe fn run() -> Result<(), String> { + let mut args = env::args().skip(1); + let s3_path = args.next().ok_or("missing S3 extension path")?; + let timeout_path = args.next().ok_or("missing Timeout extension path")?; + if args.next().is_some() { + return Err("expected exactly two extension paths".to_string()); + } + + let s3 = unsafe { Library::new(&s3_path) }.map_err(|err| err.to_string())?; + let create: Symbol<'_, CreateOperator> = unsafe { + s3.get(b"opendal_service_s3_bootstrap_v1\0") + .map_err(|err| err.to_string())? + }; + let operator = unsafe { create() }; + if operator.is_null() { + return Err("S3 extension failed to create an operator".to_string()); + } + + let timeout = unsafe { Library::new(&timeout_path) }.map_err(|err| err.to_string())?; + let apply: Symbol<'_, ApplyLayer> = unsafe { + timeout + .get(b"opendal_layer_timeout_bootstrap_v1\0") + .map_err(|err| err.to_string())? + }; + let operator = unsafe { apply(operator) }; + if operator.is_null() { + return Err("Timeout extension failed to apply its layer".to_string()); + } + + let operator = unsafe { Box::from_raw(operator.cast::()) }; + let info = operator.info(); + println!( + "scheme={} name={} root={}", + info.scheme(), + info.name(), + info.root() + ); + + // Drop the operator before either library so its service and layer vtables + // still point to loaded code. + drop(operator); + drop(timeout); + drop(s3); + Ok(()) +} diff --git a/bindings/docs/dynamic-extensions/prototype/run-linux.sh b/bindings/docs/dynamic-extensions/prototype/run-linux.sh new file mode 100755 index 000000000000..578aed7e14bd --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/run-linux.sh @@ -0,0 +1,27 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +set -eu + +prototype_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +s3_target="$prototype_dir/target/s3" +timeout_target="$prototype_dir/target/timeout" +host_target="$prototype_dir/target/host" +s3="$s3_target/release/libs3_extension.so" +timeout="$timeout_target/release/libtimeout_extension.so" +host="$host_target/release/host" + +CARGO_TARGET_DIR="$s3_target" cargo build --release --locked \ + --manifest-path "$prototype_dir/Cargo.toml" --package s3-extension +CARGO_TARGET_DIR="$timeout_target" cargo build --release --locked \ + --manifest-path "$prototype_dir/Cargo.toml" --package timeout-extension +CARGO_TARGET_DIR="$host_target" cargo build --release --locked \ + --manifest-path "$prototype_dir/Cargo.toml" --package host +"$prototype_dir/audit-elf-exports.sh" \ + "$s3" opendal_service_s3_bootstrap_v1 +"$prototype_dir/audit-elf-exports.sh" \ + "$timeout" opendal_layer_timeout_bootstrap_v1 +"$host" "$s3" "$timeout" diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml new file mode 100644 index 000000000000..b7e7fe639842 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +[package] +name = "s3-extension" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core.workspace = true +opendal-service-s3.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs b/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs new file mode 100644 index 000000000000..301a4530d4aa --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs @@ -0,0 +1,20 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map b/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map new file mode 100644 index 000000000000..cbc830d1afe0 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map @@ -0,0 +1,11 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +OPENDAL_S3_EXTENSION_1 { + global: + opendal_service_s3_bootstrap_v1; + local: + *; +}; diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs b/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs new file mode 100644 index 000000000000..fc69ebe8633a --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs @@ -0,0 +1,25 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0. + +use std::ffi::c_void; +use std::panic::{AssertUnwindSafe, catch_unwind}; + +use opendal_core::Operator; +use opendal_service_s3::S3; + +#[unsafe(no_mangle)] +pub extern "C" fn opendal_service_s3_bootstrap_v1() -> *mut c_void { + catch_unwind(AssertUnwindSafe(|| { + match Operator::new(S3::default().bucket("prototype-bucket").region("us-east-1")) { + Ok(operator) => Box::into_raw(Box::new(operator)).cast::(), + Err(err) => { + eprintln!("failed to construct prototype S3 operator: {err}"); + std::ptr::null_mut() + } + } + })) + .unwrap_or_default() +} diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml new file mode 100644 index 000000000000..788fdc7ed441 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml @@ -0,0 +1,19 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +[package] +name = "timeout-extension" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core.workspace = true +opendal-layer-timeout.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs b/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs new file mode 100644 index 000000000000..301a4530d4aa --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs @@ -0,0 +1,20 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map b/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map new file mode 100644 index 000000000000..5fec693a47a7 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map @@ -0,0 +1,11 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0. + +OPENDAL_TIMEOUT_EXTENSION_1 { + global: + opendal_layer_timeout_bootstrap_v1; + local: + *; +}; diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs b/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs new file mode 100644 index 000000000000..7dac08708ef7 --- /dev/null +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs @@ -0,0 +1,30 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0. + +use std::ffi::c_void; +use std::panic::{AssertUnwindSafe, catch_unwind}; + +use opendal_core::Operator; +use opendal_layer_timeout::TimeoutLayer; + +#[unsafe(no_mangle)] +/// Applies `TimeoutLayer` to an operator from the exact prototype build. +/// +/// # Safety +/// +/// `operator` must be a non-null pointer returned by the S3 extension from the +/// same compiler, target, profile, flags, source checkout, and lockfile. +pub unsafe extern "C" fn opendal_layer_timeout_bootstrap_v1(operator: *mut c_void) -> *mut c_void { + if operator.is_null() { + return std::ptr::null_mut(); + } + + catch_unwind(AssertUnwindSafe(|| { + let operator = unsafe { Box::from_raw(operator.cast::()) }; + Box::into_raw(Box::new(operator.layer(TimeoutLayer::default()))).cast::() + })) + .unwrap_or_default() +} diff --git a/bindings/docs/dynamic-extensions/symbol-isolation.md b/bindings/docs/dynamic-extensions/symbol-isolation.md new file mode 100644 index 000000000000..5220e88b4472 --- /dev/null +++ b/bindings/docs/dynamic-extensions/symbol-isolation.md @@ -0,0 +1,176 @@ + + +# Native Extension Symbol Isolation + +Status: pre-RFC contract for the shared runtime candidate. OpenDAL does not +provide or guarantee this isolation today. + +Each final native artifact statically contains much of its Rust dependency +graph. Loading two artifacts into one process can therefore create two copies +of `opendal-core`, Tokio, and other dependencies. Matching source versions do +not turn those copies into one runtime, and Rust visibility does not determine +which symbols a platform dynamic loader exports. + +The shared runtime design requires two separate properties: + +1. **Symbol isolation** prevents one native artifact from accidentally + resolving private code or data from another artifact. +2. **Runtime state ownership** gives process-coupled facilities one explicit + owner and exposes them to extensions through runtime handles or SDK + functions. + +Symbol isolation makes private dependency copies possible. It does not make +duplicated runtime state safe. + +## Isolation Invariants + +The runtime and every native extension must satisfy these invariants: + +- The runtime exports only its versioned runtime bootstrap and explicitly + documented platform integration symbols. +- A service or layer extension exports only its generated, package-unique + bootstrap symbol and any initializer required by its language runtime. +- Every other package symbol has local or hidden visibility in the final + artifact. +- An artifact does not rely on unresolved Rust symbols being supplied by the + main binding, runtime, or another extension. +- A private dependency type, vtable, allocator-owned value, thread-local, or + mutable global does not cross the extension boundary. +- The side that creates an allocation destroys it unless an SDK handle + explicitly transfers ownership. +- The runtime pins an extension while any callback, vtable, task, or object can + execute code from that extension. + +An exact OpenDAL release match does not relax these rules. Release matching +selects the release-specific SDK contract; it does not authorize ambient +symbol resolution between artifacts. + +## Dependency Ownership + + + +| Dependency kind | Examples | Required treatment | +| --------------- | -------- | ------------------ | +| Runtime-owned state | Operator registry, executor, timers, shared HTTP context, library leases | The shared runtime owns the state. Extensions use runtime handles or SDK functions. | +| Extension-private implementation | Signing, XML parsing, checksums, service-specific caches | The extension may contain a private copy when its symbols stay local and its values do not cross the boundary. | +| Process-global native facility | JVM, TLS libraries with global configuration, native client libraries | Packaging selects one compatible process-wide instance or isolates the facility out of process. Symbol hiding alone is insufficient. | +| Exact-release interface | Factory adapters, opaque handle operations, creator-side destructors | The extension SDK generates the interface for one coordinated OpenDAL release. | +| Language initializer | CPython, Ruby, or Node-API module entry | The artifact exports only the initializer required by that language in addition to any explicitly selected OpenDAL bootstrap. | + + + +This classification applies to every transitive dependency, not only Tokio. +The SDK must decide whether a dependency is runtime-owned, extension-private, +or process-global before its code enters an extension artifact. + +## Export Surfaces + +The intended native export surfaces are small and reviewable. For example: + +```text +shared runtime: + opendal_runtime_get_api_v1 + +S3 extension: + opendal_service_s3_bootstrap_v1 + +Timeout extension: + opendal_layer_timeout_bootstrap_v1 +``` + +A language-native addon may instead expose the initializer required by its +language runtime. For example, Node-API controls the addon initializer name. +The initializer must return or register the same validated package metadata +and API rather than creating an unversioned second interface. + +Public Rust items such as `Operator`, `Layer`, or Tokio functions are source +interfaces. They are not part of the dynamic export allowlist and do not form a +stable ABI. + +## Platform Enforcement + +The SDK build pipeline owns symbol visibility at the final link step: + +- On ELF targets, it uses a version script or equivalent export list, localizes + archive symbols, and inspects both dynamic symbols and `DT_NEEDED` entries. + `RTLD_LOCAL` remains defense in depth; it is not the isolation mechanism. +- On macOS, it uses an exported-symbols list and inspects exports, imports, and + install names in the final Mach-O artifact. +- On Windows, it generates an explicit `.def` file or equivalent export list + and inspects the PE export/import tables and dependent DLLs. + +The build must fail when the toolchain cannot enforce the selected export +surface. "Where supported" is not sufficient for an official extension target. + +## Artifact Monitoring + +Every release pipeline produces a normalized report for the runtime, each main +binding artifact, and each official extension artifact. The report records: + +- Exported dynamic symbols. +- Imported dynamic symbols and their provider libraries. +- Direct native library dependencies. +- Target identity and the SDK inputs that selected the allowlists. + +The pipeline compares the report with a checked-in or generated allowlist and +fails on any unreviewed addition. This catches transitive crates that introduce +`no_mangle` or C exports even when their Rust symbols remain hidden. Release +artifacts retain the report so later toolchain or dependency updates can be +compared with the accepted baseline. + +Artifact inspection must cover the final repaired wheel, gem, npm package, or +shared runtime artifact. Auditing an intermediate Cargo output does not detect +changes introduced by packaging repair, symbol stripping, or native dependency +relocation. + +## Co-loading Tests + +Export inspection cannot prove runtime behavior. The conformance suite also +loads multiple independently compiled extensions into one process and verifies: + +1. Each loader lookup resolves only the package-unique bootstrap requested by + the manifest. +2. No deliberately duplicated private sentinel resolves across artifacts. +3. An operator created by a service extension composes with a layer extension + through the exact-release SDK interface. +4. Real operations exercise runtime-coupled facilities such as timers, + executors, credential file loading, HTTP, and cancellation. +5. All extension-owned objects are destroyed before a test unloads a library. + +The [prototype](prototype/README.md) supplies an initial Linux export audit and +explores layout-sensitive `Operator` pointer exchange between independently +linked S3 and Timeout artifacts. It does not implement the proposed extension +SDK, its ownership contract, or a shared runtime. Therefore it does not satisfy +the third, fourth, or fifth conformance properties. + +## Non-goals + +Symbol isolation does not provide: + +- A security sandbox for native extensions. +- A stable Rust ABI across OpenDAL releases. +- Compatibility between arbitrary process-global native libraries. +- Shared state merely because two artifacts contain the same dependency + version. +- Safe library unloading without the lifetime rules in the compatibility + contract. + +Native extensions remain trusted in-process code under the project +[security threat model](../../../SECURITY-THREAT-MODEL.md). The isolation policy +prevents accidental linking and interposition; it does not constrain malicious +native code. diff --git a/core/services/s3/Cargo.toml b/core/services/s3/Cargo.toml index 7512b310a530..d747fd7de648 100644 --- a/core/services/s3/Cargo.toml +++ b/core/services/s3/Cargo.toml @@ -34,7 +34,7 @@ all-features = true [dependencies] base64 = { workspace = true } bytes = { workspace = true } -crc-fast = "1.9.0" +crc-fast = { version = "1.9.0", default-features = false, features = ["std"] } http = { workspace = true } log = { workspace = true } md-5 = "0.11.0" From 3ca9091478344bdcf505097cfa27398916c84228 Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Sun, 2 Aug 2026 21:51:42 +0800 Subject: [PATCH 6/8] feat(python): prototype dynamic service packages --- .../dynamic-extensions/prototype/Cargo.toml | 14 +- .../prototype/audit-elf-exports.sh | 14 +- .../prototype/host/Cargo.toml | 16 +- .../prototype/host/src/main.rs | 19 +- .../dynamic-extensions/prototype/run-linux.sh | 14 +- .../prototype/s3-extension/Cargo.toml | 16 +- .../prototype/s3-extension/build.rs | 19 +- .../prototype/s3-extension/exports.map | 14 +- .../prototype/s3-extension/src/lib.rs | 19 +- .../prototype/timeout-extension/Cargo.toml | 16 +- .../prototype/timeout-extension/build.rs | 19 +- .../prototype/timeout-extension/exports.map | 14 +- .../prototype/timeout-extension/src/lib.rs | 19 +- .../dynamic-extensions/.cargo/config.toml | 17 + bindings/python/dynamic-extensions/.gitignore | 16 + bindings/python/dynamic-extensions/Cargo.lock | 1313 +++++++++++++++++ bindings/python/dynamic-extensions/Cargo.toml | 33 + bindings/python/dynamic-extensions/README.md | 92 ++ .../dynamic-extensions/audit-elf-exports.sh | 41 + .../dynamic-extensions/python/example.py | 29 + .../python/main/opendal/__init__.py | 22 + .../python/main/opendal/_runtime.py | 304 ++++ .../python/main/opendal/services/__init__.py | 18 + .../opendal/services/fs/__init__.py | 27 + .../opendal/services/s3/__init__.py | 27 + .../dynamic-extensions/python/test_poc.py | 71 + .../dynamic-extensions/run-python-linux.sh | 54 + .../dynamic-extensions/runtime/Cargo.toml | 30 + .../dynamic-extensions/runtime/build.rs | 29 + .../dynamic-extensions/runtime/exports.map | 21 + .../dynamic-extensions/runtime/src/lib.rs | 397 +++++ .../python/dynamic-extensions/sdk/Cargo.toml | 22 + .../python/dynamic-extensions/sdk/src/lib.rs | 224 +++ .../dynamic-extensions/service-fs/Cargo.toml | 32 + .../dynamic-extensions/service-fs/build.rs | 29 + .../dynamic-extensions/service-fs/exports.map | 21 + .../dynamic-extensions/service-fs/src/lib.rs | 218 +++ .../dynamic-extensions/service-s3/Cargo.toml | 32 + .../dynamic-extensions/service-s3/build.rs | 29 + .../dynamic-extensions/service-s3/exports.map | 21 + .../dynamic-extensions/service-s3/src/lib.rs | 222 +++ 41 files changed, 3560 insertions(+), 44 deletions(-) create mode 100644 bindings/python/dynamic-extensions/.cargo/config.toml create mode 100644 bindings/python/dynamic-extensions/.gitignore create mode 100644 bindings/python/dynamic-extensions/Cargo.lock create mode 100644 bindings/python/dynamic-extensions/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/README.md create mode 100755 bindings/python/dynamic-extensions/audit-elf-exports.sh create mode 100644 bindings/python/dynamic-extensions/python/example.py create mode 100644 bindings/python/dynamic-extensions/python/main/opendal/__init__.py create mode 100644 bindings/python/dynamic-extensions/python/main/opendal/_runtime.py create mode 100644 bindings/python/dynamic-extensions/python/main/opendal/services/__init__.py create mode 100644 bindings/python/dynamic-extensions/python/service-fs/opendal/services/fs/__init__.py create mode 100644 bindings/python/dynamic-extensions/python/service-s3/opendal/services/s3/__init__.py create mode 100644 bindings/python/dynamic-extensions/python/test_poc.py create mode 100755 bindings/python/dynamic-extensions/run-python-linux.sh create mode 100644 bindings/python/dynamic-extensions/runtime/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/runtime/build.rs create mode 100644 bindings/python/dynamic-extensions/runtime/exports.map create mode 100644 bindings/python/dynamic-extensions/runtime/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/sdk/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/sdk/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/service-fs/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/service-fs/build.rs create mode 100644 bindings/python/dynamic-extensions/service-fs/exports.map create mode 100644 bindings/python/dynamic-extensions/service-fs/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/service-s3/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/service-s3/build.rs create mode 100644 bindings/python/dynamic-extensions/service-s3/exports.map create mode 100644 bindings/python/dynamic-extensions/service-s3/src/lib.rs diff --git a/bindings/docs/dynamic-extensions/prototype/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/Cargo.toml index be712fd7dcaa..7fc5619917e6 100644 --- a/bindings/docs/dynamic-extensions/prototype/Cargo.toml +++ b/bindings/docs/dynamic-extensions/prototype/Cargo.toml @@ -1,7 +1,17 @@ # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. [workspace] members = ["host", "s3-extension", "timeout-extension"] diff --git a/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh b/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh index 94fda6c932a4..59a84bbbd1ab 100755 --- a/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh +++ b/bindings/docs/dynamic-extensions/prototype/audit-elf-exports.sh @@ -1,8 +1,18 @@ #!/usr/bin/env sh # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. set -eu diff --git a/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml index a6e1f425291c..a49573dba213 100644 --- a/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml +++ b/bindings/docs/dynamic-extensions/prototype/host/Cargo.toml @@ -1,12 +1,22 @@ # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. [package] -name = "host" edition.workspace = true license.workspace = true +name = "host" publish.workspace = true rust-version.workspace = true version.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/host/src/main.rs b/bindings/docs/dynamic-extensions/prototype/host/src/main.rs index 6fb57c6eb087..303d1aa45d97 100644 --- a/bindings/docs/dynamic-extensions/prototype/host/src/main.rs +++ b/bindings/docs/dynamic-extensions/prototype/host/src/main.rs @@ -1,8 +1,17 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0. +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. use std::env; use std::ffi::c_void; diff --git a/bindings/docs/dynamic-extensions/prototype/run-linux.sh b/bindings/docs/dynamic-extensions/prototype/run-linux.sh index 578aed7e14bd..7283b8d85420 100755 --- a/bindings/docs/dynamic-extensions/prototype/run-linux.sh +++ b/bindings/docs/dynamic-extensions/prototype/run-linux.sh @@ -1,8 +1,18 @@ #!/usr/bin/env sh # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. set -eu diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml index b7e7fe639842..6a30fb418d6b 100644 --- a/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/Cargo.toml @@ -1,12 +1,22 @@ # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. [package] -name = "s3-extension" edition.workspace = true license.workspace = true +name = "s3-extension" publish.workspace = true rust-version.workspace = true version.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs b/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs index 301a4530d4aa..501b9f44298a 100644 --- a/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/build.rs @@ -1,8 +1,17 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0. +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. use std::env; use std::path::PathBuf; diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map b/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map index cbc830d1afe0..09035fa37b48 100644 --- a/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/exports.map @@ -1,7 +1,17 @@ # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. OPENDAL_S3_EXTENSION_1 { global: diff --git a/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs b/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs index fc69ebe8633a..70744aa151bc 100644 --- a/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs +++ b/bindings/docs/dynamic-extensions/prototype/s3-extension/src/lib.rs @@ -1,8 +1,17 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0. +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. use std::ffi::c_void; use std::panic::{AssertUnwindSafe, catch_unwind}; diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml b/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml index 788fdc7ed441..a8673d4640f9 100644 --- a/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/Cargo.toml @@ -1,12 +1,22 @@ # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. [package] -name = "timeout-extension" edition.workspace = true license.workspace = true +name = "timeout-extension" publish.workspace = true rust-version.workspace = true version.workspace = true diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs b/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs index 301a4530d4aa..501b9f44298a 100644 --- a/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/build.rs @@ -1,8 +1,17 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0. +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. use std::env; use std::path::PathBuf; diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map b/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map index 5fec693a47a7..d6cca9c4d8c2 100644 --- a/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/exports.map @@ -1,7 +1,17 @@ # Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with +# contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. -# The ASF licenses this file to you under the Apache License, Version 2.0. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. OPENDAL_TIMEOUT_EXTENSION_1 { global: diff --git a/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs b/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs index 7dac08708ef7..0828cbfabb93 100644 --- a/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs +++ b/bindings/docs/dynamic-extensions/prototype/timeout-extension/src/lib.rs @@ -1,8 +1,17 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0. +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. use std::ffi::c_void; use std::panic::{AssertUnwindSafe, catch_unwind}; diff --git a/bindings/python/dynamic-extensions/.cargo/config.toml b/bindings/python/dynamic-extensions/.cargo/config.toml new file mode 100644 index 000000000000..75f24ae96dae --- /dev/null +++ b/bindings/python/dynamic-extensions/.cargo/config.toml @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[target.'cfg(target_os = "linux")'] +rustflags = ["-C", "link-arg=-Wl,--exclude-libs,ALL"] diff --git a/bindings/python/dynamic-extensions/.gitignore b/bindings/python/dynamic-extensions/.gitignore new file mode 100644 index 000000000000..d7d2adf88a85 --- /dev/null +++ b/bindings/python/dynamic-extensions/.gitignore @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +/target/ diff --git a/bindings/python/dynamic-extensions/Cargo.lock b/bindings/python/dynamic-extensions/Cargo.lock new file mode 100644 index 000000000000..295521b82f6d --- /dev/null +++ b/bindings/python/dynamic-extensions/Cargo.lock @@ -0,0 +1,1313 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "base64" +version = "0.23.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b25655df2c3cdd83c5e5b293b88acd880332b2ddadd7c30ac43144fdc0033da9" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "cmov" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "const-random" +version = "0.1.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" +dependencies = [ + "const-random-macro", +] + +[[package]] +name = "const-random-macro" +version = "0.1.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" +dependencies = [ + "getrandom 0.2.17", + "once_cell", + "tiny-keccak", +] + +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + +[[package]] +name = "crc-fast" +version = "1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e75b2483e97a5a7da73ac68a05b629f9c53cff58d8ed1c77866079e18b00dba5" +dependencies = [ + "digest 0.10.7", + "spin", +] + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "crypto-common" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" +dependencies = [ + "generic-array", + "typenum", +] + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "ctutils" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d5515a3834141de9eafb9717ad39eea8247b5674e6066c404e8c4b365d2a29e" +dependencies = [ + "cmov", +] + +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror", +] + +[[package]] +name = "digest" +version = "0.10.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" +dependencies = [ + "crypto-common 0.1.7", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer", + "const-oid", + "crypto-common 0.2.2", + "ctutils", +] + +[[package]] +name = "displaydoc" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "dlv-list" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "442039f5147480ba31067cb00ada1adae6892028e40e45fc5de7b7df6dcc1b5f" +dependencies = [ + "const-random", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "fs-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-dynamic-extension-sdk", + "opendal-service-fs", + "serde_json", + "tokio", +] + +[[package]] +name = "futures" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a88cf1f829d945f548cf8fec32c61b1f202b6d93b45848602fc02af4b12ad218" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-executor" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6754879cc9f2c66f88c6e5c35344bb0bdb0708b0352b1201815667c7eabc7458" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "generic-array" +version = "0.14.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" +dependencies = [ + "typenum", + "version_check", +] + +[[package]] +name = "getrandom" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" +dependencies = [ + "cfg-if", + "libc", + "wasi", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "hashbrown" +version = "0.14.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" + +[[package]] +name = "hex" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" + +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.3", +] + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "hybrid-array" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" +dependencies = [ + "typenum", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +dependencies = [ + "defmt", +] + +[[package]] +name = "jiff-static" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.3", +] + +[[package]] +name = "mea" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2640d335e7273dacdcf51044026139b2e269c3bb0dfc3f8cb3496b85e3f6a42c" +dependencies = [ + "slab", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opendal-core" +version = "0.58.1" +dependencies = [ + "anyhow", + "base64 0.23.0", + "bytes", + "futures", + "http", + "jiff", + "log", + "md-5", + "mea", + "percent-encoding", + "quick-xml", + "reqsign-core", + "serde", + "serde_json", + "tokio", + "url", + "uuid", + "web-time", +] + +[[package]] +name = "opendal-dynamic-extension-sdk" +version = "0.0.0" + +[[package]] +name = "opendal-runtime-poc" +version = "0.0.0" +dependencies = [ + "libloading", + "opendal-core", + "opendal-dynamic-extension-sdk", +] + +[[package]] +name = "opendal-service-fs" +version = "0.58.1" +dependencies = [ + "bytes", + "log", + "opendal-core", + "serde", + "tokio", + "xattr", +] + +[[package]] +name = "opendal-service-s3" +version = "0.58.1" +dependencies = [ + "base64 0.23.0", + "bytes", + "crc-fast", + "http", + "log", + "md-5", + "opendal-core", + "quick-xml", + "reqsign-aws-v4", + "reqsign-core", + "reqsign-file-read-tokio", + "serde", + "url", +] + +[[package]] +name = "ordered-multimap" +version = "0.7.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "49203cdcae0030493bad186b28da2fa25645fa276a51b6fec8010d281e02ef79" +dependencies = [ + "dlv-list", + "hashbrown", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "reqsign-aws-v4" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cc883bc56889f3e4a419265c87facea222a921debc5c6f15c7fd8b68ec4b36b2" +dependencies = [ + "anyhow", + "bytes", + "form_urlencoded", + "hex", + "http", + "log", + "percent-encoding", + "quick-xml", + "reqsign-core", + "rust-ini", + "serde", + "serde_json", + "serde_urlencoded", + "sha1", +] + +[[package]] +name = "reqsign-core" +version = "3.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e38b44697c60a823705ccef85cb04d8e0527c9d16ed7c58bf1c6395bdd24ceb" +dependencies = [ + "anyhow", + "base64 0.22.1", + "bytes", + "futures", + "hex", + "hmac", + "http", + "jiff", + "log", + "percent-encoding", + "sha1", + "sha2", + "windows-sys", +] + +[[package]] +name = "reqsign-file-read-tokio" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "688ff0ae421b8d4b92b53fdafaf53df2de28f428a9962edcf21702990b26f74b" +dependencies = [ + "anyhow", + "reqsign-core", + "tokio", +] + +[[package]] +name = "rust-ini" +version = "0.21.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "796e8d2b6696392a43bea58116b667fb4c29727dc5abd27d6acf338bb4f688c7" +dependencies = [ + "cfg-if", + "ordered-multimap", +] + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags 2.13.1", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "ryu" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" + +[[package]] +name = "s3-extension" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-dynamic-extension-sdk", + "opendal-service-s3", + "serde_json", + "tokio", +] + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_urlencoded" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" +dependencies = [ + "form_urlencoded", + "itoa", + "ryu", + "serde", +] + +[[package]] +name = "sha1" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures", + "digest 0.11.3", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "spin" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3" + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tiny-keccak" +version = "2.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" +dependencies = [ + "crunchy", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "pin-project-lite", + "tokio-macros", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" +dependencies = [ + "getrandom 0.4.3", + "js-sys", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "wasi" +version = "0.11.1+wasi-snapshot-preview1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/bindings/python/dynamic-extensions/Cargo.toml b/bindings/python/dynamic-extensions/Cargo.toml new file mode 100644 index 000000000000..5d2c054e2f96 --- /dev/null +++ b/bindings/python/dynamic-extensions/Cargo.toml @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +members = ["runtime", "sdk", "service-fs", "service-s3"] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" + +[workspace.dependencies] +libloading = "0.8.9" +opendal-core = { path = "../../../core/core", default-features = false } +opendal-dynamic-extension-sdk = { path = "sdk" } +opendal-service-fs = { path = "../../../core/services/fs" } +opendal-service-s3 = { path = "../../../core/services/s3" } +tokio = { version = "1", features = ["rt-multi-thread", "time"] } diff --git a/bindings/python/dynamic-extensions/README.md b/bindings/python/dynamic-extensions/README.md new file mode 100644 index 000000000000..37ad40caa03e --- /dev/null +++ b/bindings/python/dynamic-extensions/README.md @@ -0,0 +1,92 @@ + + +# Python Dynamic Extension POC + +This in-tree POC tests the package and loading model proposed in +[`bindings/docs/dynamic-extensions/python.md`](../../docs/dynamic-extensions/python.md). +It does not change the released Python binding. + +The workspace builds three independently linked native artifacts: + +- `opendal-runtime-poc` owns protocol negotiation, service registration, + opaque Python operator handles, library leases, and error transport. +- `s3-extension` owns the S3 builder, operator, dependencies, and operations. +- `fs-extension` owns the FS builder, operator, dependencies, and operations. + +The Python package roots model three separately installed distributions. The +main package exposes `opendal.Operator`. Importing `opendal.services.s3` or +`opendal.services.fs` resolves that package's native artifact and registers its +logical manifest without loading native service code. The manifest does not +contain an artifact path or target identity, and configuration crosses the +runtime interface as string pairs. + +On Linux, run: + +```console +./run-python-linux.sh +``` + +The separately staged packages support this usage: + +```python +import opendal.services.fs +import opendal.services.s3 +from opendal import Operator + +with Operator("s3", bucket="my-bucket", region="us-east-1") as s3: + print(s3.info) + +with Operator("fs", root="/tmp/opendal") as fs: + fs.write("hello.txt", b"Hello, OpenDAL!") + print(fs.read("hello.txt")) +``` + +The runner executes the complete [`python/example.py`](python/example.py) in +addition to the assertions in `python/test_poc.py`. + +The script builds each native artifact in a separate Cargo target directory, +checks the final ELF export allowlists, stages the three Python package roots, +and verifies these behaviors: + +1. S3 construction fails before importing its service package. +2. Importing the S3 package registers metadata without loading native code. +3. Constructing an S3 operator loads its package-unique bootstrap and validates + its package, component, entry symbol, protocol, and OpenDAL identities. +4. Importing the FS package registers FS and completes a real write and read. +5. Closing a Python operator invokes the extension-provided destructor before + releasing its library lease. + +## Deliberate Gaps + +This POC keeps the interface small enough to answer the Python packaging +question. It exposes only construction, information, read, write, and +destruction. Each extension owns the Tokio runtime used by its operations. +Consequently, it does not yet prove the selected shared-runtime design's most +important property: one OpenDAL and Tokio graph that can compose arbitrary +native layers. + +An earlier iteration transferred independently linked `Operator` values into a +runtime-owned Tokio graph. A real FS operation aborted because the FS library's +Tokio thread-local state could not observe the runtime library's Tokio context. +Another iteration used a Rust `dylib`, but separate S3 and FS builds generated +different runtime binaries because downstream monomorphizations changed the +dylib. The production design must solve this linkage problem or accept a wider +operation interface before extracting layers. + +The POC also omits wheel building, automatic entry-point discovery, async +Python operations, aliases, typed configuration, and non-Linux targets. diff --git a/bindings/python/dynamic-extensions/audit-elf-exports.sh b/bindings/python/dynamic-extensions/audit-elf-exports.sh new file mode 100755 index 000000000000..59a84bbbd1ab --- /dev/null +++ b/bindings/python/dynamic-extensions/audit-elf-exports.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +if [ "$#" -ne 2 ]; then + echo "usage: $0 ARTIFACT EXPECTED_SYMBOL" >&2 + exit 2 +fi + +artifact=$1 +expected=$2 +actual=$( + nm --dynamic --defined-only --extern-only --format=posix "$artifact" \ + | awk '{ print $1 }' \ + | sed 's/@.*//' \ + | LC_ALL=C sort -u +) + +if [ "$actual" != "$expected" ]; then + echo "unexpected exports in $artifact" >&2 + echo "expected: $expected" >&2 + echo "actual:" >&2 + echo "$actual" >&2 + exit 1 +fi + +echo "$artifact exports only $expected" diff --git a/bindings/python/dynamic-extensions/python/example.py b/bindings/python/dynamic-extensions/python/example.py new file mode 100644 index 000000000000..e946f993ca73 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/example.py @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import tempfile + +import opendal.services.fs +import opendal.services.s3 +from opendal import Operator + + +with Operator("s3", bucket="my-bucket", region="us-east-1") as s3: + print(s3.info) + +with tempfile.TemporaryDirectory() as root: + with Operator("fs", root=root) as fs: + fs.write("hello.txt", b"Hello, OpenDAL!") + print(fs.read("hello.txt")) diff --git a/bindings/python/dynamic-extensions/python/main/opendal/__init__.py b/bindings/python/dynamic-extensions/python/main/opendal/__init__.py new file mode 100644 index 000000000000..2aa5d0941cc8 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/main/opendal/__init__.py @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pkgutil import extend_path + +__path__ = extend_path(__path__, __name__) + +from ._runtime import Operator, RuntimeProtocolError + +__all__ = ["Operator", "RuntimeProtocolError"] diff --git a/bindings/python/dynamic-extensions/python/main/opendal/_runtime.py b/bindings/python/dynamic-extensions/python/main/opendal/_runtime.py new file mode 100644 index 000000000000..21b46d50e1ea --- /dev/null +++ b/bindings/python/dynamic-extensions/python/main/opendal/_runtime.py @@ -0,0 +1,304 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import ctypes +import json +import threading +from collections.abc import Mapping +from pathlib import Path +from typing import Any + +RUNTIME_PROTOCOL = 1 +STATUS_OK = 0 +STATUS_BUFFER_TOO_SMALL = 6 + + +class RuntimeProtocolError(RuntimeError): + pass + + +class ByteSlice(ctypes.Structure): + _fields_ = [("data", ctypes.c_void_p), ("len", ctypes.c_size_t)] + + +class KeyValue(ctypes.Structure): + _fields_ = [("key", ByteSlice), ("value", ByteSlice)] + + +class OutputBuffer(ctypes.Structure): + _fields_ = [ + ("data", ctypes.c_void_p), + ("capacity", ctypes.c_size_t), + ("len", ctypes.c_size_t), + ] + + +class ServiceRegistrationV1(ctypes.Structure): + _fields_ = [ + ("struct_size", ctypes.c_size_t), + ("required_runtime_protocol", ctypes.c_uint32), + ("package_id", ByteSlice), + ("component_id", ByteSlice), + ("entry_symbol", ByteSlice), + ("library_path", ByteSlice), + ] + + +class RuntimeProtocolInfoV1(ctypes.Structure): + _fields_ = [ + ("struct_size", ctypes.c_size_t), + ("minimum_runtime_protocol", ctypes.c_uint32), + ("runtime_protocol", ctypes.c_uint32), + ] + + +RegisterServiceFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.POINTER(ServiceRegistrationV1), + ctypes.POINTER(OutputBuffer), +) +CreateOperatorFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ByteSlice, + ctypes.POINTER(KeyValue), + ctypes.c_size_t, + ctypes.POINTER(ctypes.c_void_p), + ctypes.POINTER(OutputBuffer), +) +OperatorInfoFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.c_void_p, + ctypes.POINTER(OutputBuffer), + ctypes.POINTER(OutputBuffer), +) +OperatorWriteFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.c_void_p, + ByteSlice, + ByteSlice, + ctypes.POINTER(OutputBuffer), +) +OperatorReadFn = ctypes.CFUNCTYPE( + ctypes.c_int32, + ctypes.c_void_p, + ByteSlice, + ctypes.POINTER(OutputBuffer), + ctypes.POINTER(OutputBuffer), +) +OperatorDestroyFn = ctypes.CFUNCTYPE(None, ctypes.c_void_p) + + +class RuntimeApiV1(ctypes.Structure): + _fields_ = [ + ("struct_size", ctypes.c_size_t), + ("register_service", RegisterServiceFn), + ("create_operator", CreateOperatorFn), + ("operator_info", OperatorInfoFn), + ("operator_write", OperatorWriteFn), + ("operator_read", OperatorReadFn), + ("operator_destroy", OperatorDestroyFn), + ] + + +def _bytes(value: str | bytes) -> tuple[ByteSlice, ctypes.Array[ctypes.c_char]]: + raw = value.encode() if isinstance(value, str) else value + storage = ctypes.create_string_buffer(raw) + return ByteSlice(ctypes.cast(storage, ctypes.c_void_p), len(raw)), storage + + +def _output(capacity: int = 4096) -> tuple[OutputBuffer, ctypes.Array[ctypes.c_char]]: + storage = ctypes.create_string_buffer(capacity) + return OutputBuffer(ctypes.cast(storage, ctypes.c_void_p), capacity, 0), storage + + +def _message(output: OutputBuffer, storage: ctypes.Array[ctypes.c_char]) -> str: + length = min(output.len, len(storage)) + return bytes(storage[:length]).decode(errors="replace") + + +_native_path = Path(__file__).parent / "_native" / "libopendal_runtime_poc.so" +_native = ctypes.CDLL(str(_native_path)) +_native.opendal_runtime_get_api_v1.argtypes = [ + ctypes.c_uint32, + ctypes.POINTER(RuntimeProtocolInfoV1), + ctypes.POINTER(ctypes.POINTER(RuntimeApiV1)), +] +_native.opendal_runtime_get_api_v1.restype = ctypes.c_int32 + +_protocol = RuntimeProtocolInfoV1( + struct_size=ctypes.sizeof(RuntimeProtocolInfoV1), + minimum_runtime_protocol=0, + runtime_protocol=0, +) +_api_pointer = ctypes.POINTER(RuntimeApiV1)() +_status = _native.opendal_runtime_get_api_v1( + RUNTIME_PROTOCOL, ctypes.byref(_protocol), ctypes.byref(_api_pointer) +) +if _status != STATUS_OK or not _api_pointer: + raise RuntimeProtocolError( + "runtime protocol negotiation failed: " + f"required={RUNTIME_PROTOCOL}, " + f"supported={_protocol.minimum_runtime_protocol}..{_protocol.runtime_protocol}" + ) +_api = _api_pointer.contents + + +def _register_service(manifest: Mapping[str, Any], library_path: Path) -> None: + component = manifest.get("component") + if not isinstance(component, Mapping) or component.get("kind") != "service": + raise RuntimeProtocolError("manifest must describe one service") + package_id = manifest.get("package_id") + component_id = component.get("id") + entry_symbol = manifest.get("native_entry_symbol") + required_protocol = manifest.get("required_runtime_protocol") + if not all(isinstance(value, str) for value in (package_id, component_id, entry_symbol)): + raise RuntimeProtocolError( + "manifest package ID, component ID, and entry symbol must be strings" + ) + if not isinstance(required_protocol, int): + raise RuntimeProtocolError("manifest runtime protocol must be an integer") + if not library_path.is_file(): + raise RuntimeProtocolError(f"native service artifact does not exist: {library_path}") + + package, package_storage = _bytes(package_id) + service, service_storage = _bytes(component_id) + entry, entry_storage = _bytes(entry_symbol) + path, path_storage = _bytes(str(library_path.resolve())) + registration = ServiceRegistrationV1( + struct_size=ctypes.sizeof(ServiceRegistrationV1), + required_runtime_protocol=required_protocol, + package_id=package, + component_id=service, + entry_symbol=entry, + library_path=path, + ) + error, error_storage = _output() + status = _api.register_service(ctypes.byref(registration), ctypes.byref(error)) + _ = (package_storage, service_storage, entry_storage, path_storage) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + + +def _stringify(value: object) -> str: + if isinstance(value, bool): + return "true" if value else "false" + if isinstance(value, (str, int, float, Path)): + return str(value) + raise TypeError(f"unsupported configuration value: {type(value).__name__}") + + +def _options(values: Mapping[str, object]) -> tuple[Any, list[Any]]: + pairs: list[KeyValue] = [] + keepalive: list[Any] = [] + for key, value in values.items(): + encoded_key, key_storage = _bytes(key) + encoded_value, value_storage = _bytes(_stringify(value)) + pairs.append(KeyValue(encoded_key, encoded_value)) + keepalive.extend((key_storage, value_storage)) + array_type = KeyValue * len(pairs) + return array_type(*pairs), keepalive + + +class Operator: + def __init__(self, scheme: str, **config: object) -> None: + self._lock = threading.RLock() + self._handle = ctypes.c_void_p() + scheme = scheme.strip().lower().replace("_", "-") + encoded_scheme, scheme_storage = _bytes(scheme) + options, option_storage = _options(config) + handle = ctypes.c_void_p() + error, error_storage = _output() + status = _api.create_operator( + encoded_scheme, + options, + len(options), + ctypes.byref(handle), + ctypes.byref(error), + ) + _ = (scheme_storage, option_storage) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + self._handle = handle + + def close(self) -> None: + with self._lock: + handle = self._handle + if handle: + _api.operator_destroy(handle) + self._handle = ctypes.c_void_p() + + def __enter__(self) -> Operator: + return self + + def __exit__(self, *_: object) -> None: + self.close() + + def __del__(self) -> None: + self.close() + + def _require_handle(self) -> ctypes.c_void_p: + if not self._handle: + raise RuntimeProtocolError("operator is closed") + return self._handle + + @property + def info(self) -> dict[str, str]: + with self._lock: + output, output_storage = _output() + error, error_storage = _output() + status = _api.operator_info( + self._require_handle(), ctypes.byref(output), ctypes.byref(error) + ) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + return json.loads(_message(output, output_storage)) + + def write(self, path: str, data: bytes) -> None: + with self._lock: + encoded_path, path_storage = _bytes(path) + encoded_data, data_storage = _bytes(data) + error, error_storage = _output() + status = _api.operator_write( + self._require_handle(), encoded_path, encoded_data, ctypes.byref(error) + ) + _ = (path_storage, data_storage) + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + + def read(self, path: str) -> bytes: + with self._lock: + encoded_path, path_storage = _bytes(path) + output, output_storage = _output() + error, error_storage = _output() + status = _api.operator_read( + self._require_handle(), + encoded_path, + ctypes.byref(output), + ctypes.byref(error), + ) + if status == STATUS_BUFFER_TOO_SMALL: + output, output_storage = _output(output.len) + status = _api.operator_read( + self._require_handle(), + encoded_path, + ctypes.byref(output), + ctypes.byref(error), + ) + _ = path_storage + if status != STATUS_OK: + raise RuntimeProtocolError(_message(error, error_storage)) + return bytes(output_storage[: output.len]) diff --git a/bindings/python/dynamic-extensions/python/main/opendal/services/__init__.py b/bindings/python/dynamic-extensions/python/main/opendal/services/__init__.py new file mode 100644 index 000000000000..42d619211cb0 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/main/opendal/services/__init__.py @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pkgutil import extend_path + +__path__ = extend_path(__path__, __name__) diff --git a/bindings/python/dynamic-extensions/python/service-fs/opendal/services/fs/__init__.py b/bindings/python/dynamic-extensions/python/service-fs/opendal/services/fs/__init__.py new file mode 100644 index 000000000000..a46e1fb6254b --- /dev/null +++ b/bindings/python/dynamic-extensions/python/service-fs/opendal/services/fs/__init__.py @@ -0,0 +1,27 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + +from opendal._runtime import _register_service + +MANIFEST = { + "required_runtime_protocol": 1, + "package_id": "opendal-service-fs-poc", + "component": {"kind": "service", "id": "fs", "aliases": ["file"]}, + "native_entry_symbol": "opendal_service_fs_bootstrap_v1", +} + +_register_service(MANIFEST, Path(__file__).parent / "_native" / "libfs_extension.so") diff --git a/bindings/python/dynamic-extensions/python/service-s3/opendal/services/s3/__init__.py b/bindings/python/dynamic-extensions/python/service-s3/opendal/services/s3/__init__.py new file mode 100644 index 000000000000..087a84b77c01 --- /dev/null +++ b/bindings/python/dynamic-extensions/python/service-s3/opendal/services/s3/__init__.py @@ -0,0 +1,27 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + +from opendal._runtime import _register_service + +MANIFEST = { + "required_runtime_protocol": 1, + "package_id": "opendal-service-s3-poc", + "component": {"kind": "service", "id": "s3", "aliases": []}, + "native_entry_symbol": "opendal_service_s3_bootstrap_v1", +} + +_register_service(MANIFEST, Path(__file__).parent / "_native" / "libs3_extension.so") diff --git a/bindings/python/dynamic-extensions/python/test_poc.py b/bindings/python/dynamic-extensions/python/test_poc.py new file mode 100644 index 000000000000..0162f4688e0c --- /dev/null +++ b/bindings/python/dynamic-extensions/python/test_poc.py @@ -0,0 +1,71 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import tempfile +from pathlib import Path + +import opendal + + +def main() -> None: + try: + opendal.Operator("s3", bucket="prototype-bucket") + except opendal.RuntimeProtocolError as err: + assert "import its package first" in str(err) + else: + raise AssertionError("S3 must not be registered before its package import") + + import opendal.services.s3 as s3 + + assert s3.MANIFEST["component"]["id"] == "s3" + s3_library = Path(s3.__file__).parent / "_native" / "libs3_extension.so" + held_library = s3_library.with_suffix(".so.held") + s3_library.rename(held_library) + try: + try: + opendal.Operator("s3", bucket="prototype-bucket") + except opendal.RuntimeProtocolError: + pass + else: + raise AssertionError("import must not load the S3 native artifact") + finally: + held_library.rename(s3_library) + + with opendal.Operator( + "s3", bucket="prototype-bucket", region="us-east-1" + ) as operator: + assert operator.info == { + "scheme": "s3", + "name": "prototype-bucket", + "root": "/", + } + + import opendal.services.fs as fs + + assert fs.MANIFEST["component"]["id"] == "fs" + with tempfile.TemporaryDirectory() as root: + with opendal.Operator("fs", root=root) as operator: + operator.write("hello.txt", b"hello from the shared runtime") + assert operator.read("hello.txt") == b"hello from the shared runtime" + assert Path(root, "hello.txt").read_bytes() == b"hello from the shared runtime" + assert operator.info["scheme"] == "fs" + + print("imported s3 and fs service packages") + print("deferred native service loading until operator construction") + print("constructed S3 operator and completed FS write/read through runtime handles") + + +if __name__ == "__main__": + main() diff --git a/bindings/python/dynamic-extensions/run-python-linux.sh b/bindings/python/dynamic-extensions/run-python-linux.sh new file mode 100755 index 000000000000..487df48a9b9b --- /dev/null +++ b/bindings/python/dynamic-extensions/run-python-linux.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +extension_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +runtime_target="$extension_dir/target/python-runtime" +s3_target="$extension_dir/target/python-s3" +fs_target="$extension_dir/target/python-fs" +stage="$extension_dir/target/python-stage" + +CARGO_TARGET_DIR="$runtime_target" cargo build --release --locked \ + --manifest-path "$extension_dir/Cargo.toml" --package opendal-runtime-poc +CARGO_TARGET_DIR="$s3_target" cargo build --release --locked \ + --manifest-path "$extension_dir/Cargo.toml" --package s3-extension +CARGO_TARGET_DIR="$fs_target" cargo build --release --locked \ + --manifest-path "$extension_dir/Cargo.toml" --package fs-extension + +runtime="$runtime_target/release/libopendal_runtime_poc.so" +s3="$s3_target/release/libs3_extension.so" +fs="$fs_target/release/libfs_extension.so" + +"$extension_dir/audit-elf-exports.sh" "$runtime" opendal_runtime_get_api_v1 +"$extension_dir/audit-elf-exports.sh" "$s3" opendal_service_s3_bootstrap_v1 +"$extension_dir/audit-elf-exports.sh" "$fs" opendal_service_fs_bootstrap_v1 + +mkdir -p \ + "$stage/main/opendal/_native" \ + "$stage/s3/opendal/services/s3/_native" \ + "$stage/fs/opendal/services/fs/_native" +cp -R "$extension_dir/python/main/." "$stage/main/" +cp -R "$extension_dir/python/service-s3/." "$stage/s3/" +cp -R "$extension_dir/python/service-fs/." "$stage/fs/" +cp "$runtime" "$stage/main/opendal/_native/" +cp "$s3" "$stage/s3/opendal/services/s3/_native/" +cp "$fs" "$stage/fs/opendal/services/fs/_native/" + +PYTHONPATH="$stage/main:$stage/s3:$stage/fs" \ + python3 "$extension_dir/python/test_poc.py" +PYTHONPATH="$stage/main:$stage/s3:$stage/fs" \ + python3 "$extension_dir/python/example.py" diff --git a/bindings/python/dynamic-extensions/runtime/Cargo.toml b/bindings/python/dynamic-extensions/runtime/Cargo.toml new file mode 100644 index 000000000000..d6390c78c55f --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/Cargo.toml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "opendal-runtime-poc" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +libloading.workspace = true +opendal-core.workspace = true +opendal-dynamic-extension-sdk.workspace = true diff --git a/bindings/python/dynamic-extensions/runtime/build.rs b/bindings/python/dynamic-extensions/runtime/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/python/dynamic-extensions/runtime/exports.map b/bindings/python/dynamic-extensions/runtime/exports.map new file mode 100644 index 000000000000..e448b1837b28 --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_RUNTIME_POC_1 { + global: + opendal_runtime_get_api_v1; + local: + *; +}; diff --git a/bindings/python/dynamic-extensions/runtime/src/lib.rs b/bindings/python/dynamic-extensions/runtime/src/lib.rs new file mode 100644 index 000000000000..87179d98d2e1 --- /dev/null +++ b/bindings/python/dynamic-extensions/runtime/src/lib.rs @@ -0,0 +1,397 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashMap; +use std::ffi::c_void; +use std::mem::size_of; +use std::panic::{AssertUnwindSafe, catch_unwind}; +use std::sync::{Arc, LazyLock, Mutex}; + +use libloading::{Library, Symbol}; +use opendal_dynamic_extension_sdk::{ + ByteSlice, CreateOperatorFn, DestroyOperatorFn, ExtensionApiV1, ExtensionBootstrapV1, KeyValue, + OperatorInfoFn, OperatorReadFn, OperatorWriteFn, OutputBuffer, RUNTIME_PROTOCOL, RuntimeApiV1, + RuntimeProtocolInfoV1, STATUS_CONFLICT, STATUS_INCOMPATIBLE, STATUS_INVALID_ARGUMENT, + STATUS_LOAD_FAILED, STATUS_OK, STATUS_OPERATION_FAILED, ServiceRegistrationV1, write_output, +}; + +struct LoadedService { + _library: Library, + create_operator: CreateOperatorFn, + destroy_operator: DestroyOperatorFn, + operator_info: OperatorInfoFn, + operator_write: OperatorWriteFn, + operator_read: OperatorReadFn, +} + +struct RegisteredService { + package_id: String, + component_id: String, + entry_symbol: String, + library_path: String, + loaded: Mutex>>, +} + +struct RuntimeOperator { + operator: *mut c_void, + service: Arc, +} + +impl Drop for RuntimeOperator { + fn drop(&mut self) { + unsafe { (self.service.destroy_operator)(self.operator) }; + } +} + +static SERVICES: LazyLock>>> = + LazyLock::new(|| Mutex::new(HashMap::new())); + +static RUNTIME_API: RuntimeApiV1 = RuntimeApiV1 { + struct_size: size_of::(), + register_service, + create_operator, + operator_info, + operator_write, + operator_read, + operator_destroy, +}; + +fn fail(error: *mut OutputBuffer, status: i32, message: impl AsRef) -> i32 { + let _ = unsafe { write_output(error, message.as_ref().as_bytes()) }; + status +} + +fn required_struct(actual: usize, name: &str) -> Result<(), String> { + if actual < size_of::() { + return Err(format!("{name} is smaller than the version 1 layout")); + } + Ok(()) +} + +unsafe fn load_service( + registration: &RegisteredService, +) -> Result, (i32, String)> { + let mut loaded = registration.loaded.lock().map_err(|_| { + ( + STATUS_OPERATION_FAILED, + "service load mutex poisoned".to_string(), + ) + })?; + if let Some(service) = loaded.as_ref() { + return Ok(Arc::clone(service)); + } + + let library = unsafe { Library::new(®istration.library_path) } + .map_err(|err| (STATUS_LOAD_FAILED, err.to_string()))?; + let bootstrap: Symbol<'_, ExtensionBootstrapV1> = + unsafe { library.get(registration.entry_symbol.as_bytes()) } + .map_err(|err| (STATUS_LOAD_FAILED, err.to_string()))?; + let mut extension = std::ptr::null(); + let status = unsafe { bootstrap(&mut extension) }; + if status != STATUS_OK || extension.is_null() { + return Err((STATUS_LOAD_FAILED, "extension bootstrap failed".to_string())); + } + let extension: &ExtensionApiV1 = unsafe { &*extension }; + required_struct::(extension.struct_size, "extension API") + .map_err(|message| (STATUS_INCOMPATIBLE, message))?; + if extension.required_runtime_protocol > RUNTIME_PROTOCOL { + return Err(( + STATUS_INCOMPATIBLE, + "extension requires a newer runtime protocol".to_string(), + )); + } + + let extension_version = unsafe { extension.opendal_version.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_version != opendal_core::raw::VERSION { + return Err(( + STATUS_INCOMPATIBLE, + format!( + "extension OpenDAL version {extension_version} does not match runtime {}", + opendal_core::raw::VERSION + ), + )); + } + let extension_package = unsafe { extension.package_id.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_package != registration.package_id { + return Err(( + STATUS_INCOMPATIBLE, + "extension package does not match registration".to_string(), + )); + } + let extension_component = unsafe { extension.component_id.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_component != registration.component_id { + return Err(( + STATUS_INCOMPATIBLE, + "extension component does not match registration".to_string(), + )); + } + let extension_entry = unsafe { extension.entry_symbol.as_str() } + .map_err(|message| (STATUS_INCOMPATIBLE, message.to_string()))?; + if extension_entry != registration.entry_symbol { + return Err(( + STATUS_INCOMPATIBLE, + "extension entry symbol does not match registration".to_string(), + )); + } + + let service = Arc::new(LoadedService { + _library: library, + create_operator: extension.create_operator, + destroy_operator: extension.destroy_operator, + operator_info: extension.operator_info, + operator_write: extension.operator_write, + operator_read: extension.operator_read, + }); + *loaded = Some(Arc::clone(&service)); + Ok(service) +} + +unsafe extern "C" fn register_service( + registration: *const ServiceRegistrationV1, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if registration.is_null() { + return fail(error, STATUS_INVALID_ARGUMENT, "registration is null"); + } + let registration = &*registration; + if let Err(message) = required_struct::( + registration.struct_size, + "service registration", + ) { + return fail(error, STATUS_INVALID_ARGUMENT, message); + } + if registration.required_runtime_protocol > RUNTIME_PROTOCOL { + return fail( + error, + STATUS_INCOMPATIBLE, + "service requires a newer runtime protocol", + ); + } + + let package_id = match registration.package_id.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "package ID is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let component_id = match registration.component_id.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "component ID is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let entry_symbol = match registration.entry_symbol.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "entry symbol is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let library_path = match registration.library_path.as_str() { + Ok(value) if !value.is_empty() => value, + Ok(_) => return fail(error, STATUS_INVALID_ARGUMENT, "library path is empty"), + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + + let mut services = SERVICES.lock().expect("service registry mutex poisoned"); + if let Some(existing) = services.get(component_id) { + if existing.package_id == package_id { + return STATUS_OK; + } + return fail( + error, + STATUS_CONFLICT, + format!( + "service {component_id} is already owned by {}", + existing.package_id + ), + ); + } + services.insert( + component_id.to_string(), + Arc::new(RegisteredService { + package_id: package_id.to_string(), + component_id: component_id.to_string(), + entry_symbol: entry_symbol.to_string(), + library_path: library_path.to_string(), + loaded: Mutex::new(None), + }), + ); + STATUS_OK + })) + .unwrap_or_else(|_| { + fail( + error, + STATUS_OPERATION_FAILED, + "service registration panicked", + ) + }) +} + +unsafe extern "C" fn create_operator( + component_id: ByteSlice, + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if operator.is_null() { + return fail(error, STATUS_INVALID_ARGUMENT, "operator output is null"); + } + *operator = std::ptr::null_mut(); + if options_len > 0 && options.is_null() { + return fail(error, STATUS_INVALID_ARGUMENT, "options pointer is null"); + } + let component_id = match component_id.as_str() { + Ok(value) => value, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + let registration = { + let services = SERVICES.lock().expect("service registry mutex poisoned"); + match services.get(component_id) { + Some(registration) => Arc::clone(registration), + None => { + return fail( + error, + STATUS_INVALID_ARGUMENT, + format!( + "service {component_id} is not registered; import its package first" + ), + ); + } + } + }; + let service = match load_service(®istration) { + Ok(service) => service, + Err((status, message)) => return fail(error, status, message), + }; + + let mut inner = std::ptr::null_mut(); + let status = (service.create_operator)(options, options_len, &mut inner, error); + if status != STATUS_OK { + return status; + } + if inner.is_null() { + return fail( + error, + STATUS_OPERATION_FAILED, + "service returned a null operator", + ); + } + *operator = Box::into_raw(Box::new(RuntimeOperator { + operator: inner, + service, + })) + .cast::(); + STATUS_OK + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator creation panicked")) +} + +unsafe fn runtime_operator<'a>(operator: *mut c_void) -> Result<&'a RuntimeOperator, &'static str> { + if operator.is_null() { + return Err("operator handle is null"); + } + Ok(unsafe { &*operator.cast::() }) +} + +unsafe extern "C" fn operator_info( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match runtime_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + (operator.service.operator_info)(operator.operator, output, error) + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator info panicked")) +} + +unsafe extern "C" fn operator_write( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match runtime_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + (operator.service.operator_write)(operator.operator, path, data, error) + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator write panicked")) +} + +unsafe extern "C" fn operator_read( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match runtime_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, STATUS_INVALID_ARGUMENT, message), + }; + (operator.service.operator_read)(operator.operator, path, output, error) + })) + .unwrap_or_else(|_| fail(error, STATUS_OPERATION_FAILED, "operator read panicked")) +} + +unsafe extern "C" fn operator_destroy(operator: *mut c_void) { + if operator.is_null() { + return; + } + let _ = catch_unwind(AssertUnwindSafe(|| unsafe { + drop(Box::from_raw(operator.cast::())); + })); +} + +#[unsafe(no_mangle)] +/// Negotiates the runtime protocol and returns the runtime function table. +/// +/// # Safety +/// +/// `protocol_info` and `api` must point to writable storage. The caller must +/// initialize `protocol_info.struct_size` before calling this function. +pub unsafe extern "C" fn opendal_runtime_get_api_v1( + required_runtime_protocol: u32, + protocol_info: *mut RuntimeProtocolInfoV1, + api: *mut *const RuntimeApiV1, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if protocol_info.is_null() || api.is_null() { + return STATUS_INVALID_ARGUMENT; + } + let protocol_info = &mut *protocol_info; + if protocol_info.struct_size < size_of::() { + return STATUS_INVALID_ARGUMENT; + } + protocol_info.minimum_runtime_protocol = RUNTIME_PROTOCOL; + protocol_info.runtime_protocol = RUNTIME_PROTOCOL; + *api = std::ptr::null(); + if required_runtime_protocol != RUNTIME_PROTOCOL { + return STATUS_INCOMPATIBLE; + } + *api = &RUNTIME_API; + STATUS_OK + })) + .unwrap_or(STATUS_OPERATION_FAILED) +} diff --git a/bindings/python/dynamic-extensions/sdk/Cargo.toml b/bindings/python/dynamic-extensions/sdk/Cargo.toml new file mode 100644 index 000000000000..af43252dcb61 --- /dev/null +++ b/bindings/python/dynamic-extensions/sdk/Cargo.toml @@ -0,0 +1,22 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "opendal-dynamic-extension-sdk" +publish.workspace = true +rust-version.workspace = true +version.workspace = true diff --git a/bindings/python/dynamic-extensions/sdk/src/lib.rs b/bindings/python/dynamic-extensions/sdk/src/lib.rs new file mode 100644 index 000000000000..0bd04710f2b8 --- /dev/null +++ b/bindings/python/dynamic-extensions/sdk/src/lib.rs @@ -0,0 +1,224 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::slice; +use std::str; + +pub const RUNTIME_PROTOCOL: u32 = 1; +pub const STATUS_OK: i32 = 0; +pub const STATUS_INVALID_ARGUMENT: i32 = 1; +pub const STATUS_INCOMPATIBLE: i32 = 2; +pub const STATUS_CONFLICT: i32 = 3; +pub const STATUS_LOAD_FAILED: i32 = 4; +pub const STATUS_OPERATION_FAILED: i32 = 5; +pub const STATUS_BUFFER_TOO_SMALL: i32 = 6; + +#[repr(C)] +#[derive(Clone, Copy)] +pub struct ByteSlice { + pub data: *const u8, + pub len: usize, +} + +// Protocol byte slices point to immutable memory whose owner must retain it +// for the documented call or table lifetime. +unsafe impl Send for ByteSlice {} +unsafe impl Sync for ByteSlice {} + +impl ByteSlice { + pub const fn from_static(value: &'static str) -> Self { + Self { + data: value.as_ptr(), + len: value.len(), + } + } + + /// # Safety + /// + /// The caller must keep `data` valid for `len` bytes for the returned + /// slice's lifetime. + pub unsafe fn as_bytes<'a>(self) -> Result<&'a [u8], &'static str> { + if self.len == 0 { + return Ok(&[]); + } + if self.data.is_null() { + return Err("byte slice has a null pointer"); + } + Ok(unsafe { slice::from_raw_parts(self.data, self.len) }) + } + + /// # Safety + /// + /// The caller must satisfy [`ByteSlice::as_bytes`]. + pub unsafe fn as_str<'a>(self) -> Result<&'a str, &'static str> { + str::from_utf8(unsafe { self.as_bytes()? }).map_err(|_| "byte slice is not valid UTF-8") + } +} + +#[repr(C)] +#[derive(Clone, Copy)] +pub struct KeyValue { + pub key: ByteSlice, + pub value: ByteSlice, +} + +#[repr(C)] +pub struct OutputBuffer { + pub data: *mut u8, + pub capacity: usize, + pub len: usize, +} + +/// Copy bytes into a caller-owned output buffer. +/// +/// The function always records the required length. It returns +/// [`STATUS_BUFFER_TOO_SMALL`] without copying when the supplied capacity is +/// insufficient. +/// +/// # Safety +/// +/// `output` must be null or point to a valid [`OutputBuffer`]. Its `data` must +/// be valid for `capacity` writable bytes when the capacity is non-zero. +pub unsafe fn write_output(output: *mut OutputBuffer, value: &[u8]) -> i32 { + if output.is_null() { + return STATUS_INVALID_ARGUMENT; + } + + let output = unsafe { &mut *output }; + output.len = value.len(); + if output.capacity < value.len() { + return STATUS_BUFFER_TOO_SMALL; + } + if value.is_empty() { + return STATUS_OK; + } + if output.data.is_null() { + return STATUS_INVALID_ARGUMENT; + } + + unsafe { std::ptr::copy(value.as_ptr(), output.data, value.len()) }; + STATUS_OK +} + +/// Creates an extension-owned operator handle. +/// +/// Input slices remain caller-owned and are valid only for the call. On +/// success, `operator` receives a non-null handle that the caller must pass +/// only to callbacks from the same extension table and destroy exactly once. +pub type CreateOperatorFn = unsafe extern "C" fn( + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32; +/// Destroys a handle returned by [`CreateOperatorFn`]. +pub type DestroyOperatorFn = unsafe extern "C" fn(operator: *mut c_void); + +/// Describes one extension for the lifetime of its loaded native library. +/// +/// The table and all byte slices in it point to immutable package-owned +/// storage. Callers must serialize operations and destruction for each handle; +/// this POC does not define concurrent callback behavior. Operation callbacks +/// borrow their input buffers for the call and report text errors through +/// caller-owned [`OutputBuffer`] values. +#[repr(C)] +pub struct ExtensionApiV1 { + pub struct_size: usize, + pub required_runtime_protocol: u32, + pub opendal_version: ByteSlice, + pub package_id: ByteSlice, + pub component_id: ByteSlice, + pub entry_symbol: ByteSlice, + pub create_operator: CreateOperatorFn, + pub destroy_operator: DestroyOperatorFn, + pub operator_info: OperatorInfoFn, + pub operator_write: OperatorWriteFn, + pub operator_read: OperatorReadFn, +} + +/// Returns an immutable extension table owned by the loaded native library. +pub type ExtensionBootstrapV1 = unsafe extern "C" fn(extension: *mut *const ExtensionApiV1) -> i32; + +/// Provides the logical manifest fields and package-resolved artifact path. +/// +/// Every byte slice is caller-owned and remains valid only for the registration +/// call. The runtime copies fields that must outlive the call and retains the +/// loaded library while any operator created from it remains alive. +#[repr(C)] +pub struct ServiceRegistrationV1 { + pub struct_size: usize, + pub required_runtime_protocol: u32, + pub package_id: ByteSlice, + pub component_id: ByteSlice, + pub entry_symbol: ByteSlice, + pub library_path: ByteSlice, +} + +/// Registers one service and writes a diagnostic into `error` on failure. +pub type RegisterServiceFn = unsafe extern "C" fn( + registration: *const ServiceRegistrationV1, + error: *mut OutputBuffer, +) -> i32; +/// Creates a runtime-owned wrapper around an extension operator handle. +pub type CreateRuntimeOperatorFn = unsafe extern "C" fn( + component_id: ByteSlice, + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32; +/// Writes JSON operator information into caller-owned output storage. +pub type OperatorInfoFn = unsafe extern "C" fn( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32; +/// Writes bytes through an operator without taking ownership of input slices. +pub type OperatorWriteFn = unsafe extern "C" fn( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32; +/// Reads bytes into caller-owned output storage. +pub type OperatorReadFn = unsafe extern "C" fn( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32; +/// Destroys a runtime operator wrapper exactly once. +pub type OperatorDestroyFn = unsafe extern "C" fn(operator: *mut c_void); + +/// Runtime callbacks available for the lifetime of the runtime library. +#[repr(C)] +pub struct RuntimeApiV1 { + pub struct_size: usize, + pub register_service: RegisterServiceFn, + pub create_operator: CreateRuntimeOperatorFn, + pub operator_info: OperatorInfoFn, + pub operator_write: OperatorWriteFn, + pub operator_read: OperatorReadFn, + pub operator_destroy: OperatorDestroyFn, +} + +/// Reports the protocol interval supported by the loaded runtime. +#[repr(C)] +pub struct RuntimeProtocolInfoV1 { + pub struct_size: usize, + pub minimum_runtime_protocol: u32, + pub runtime_protocol: u32, +} diff --git a/bindings/python/dynamic-extensions/service-fs/Cargo.toml b/bindings/python/dynamic-extensions/service-fs/Cargo.toml new file mode 100644 index 000000000000..12ed498bc352 --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "fs-extension" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core = { workspace = true, features = ["blocking"] } +opendal-dynamic-extension-sdk.workspace = true +opendal-service-fs.workspace = true +serde_json = "1" +tokio.workspace = true diff --git a/bindings/python/dynamic-extensions/service-fs/build.rs b/bindings/python/dynamic-extensions/service-fs/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/python/dynamic-extensions/service-fs/exports.map b/bindings/python/dynamic-extensions/service-fs/exports.map new file mode 100644 index 000000000000..eaadbb8b536b --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_FS_EXTENSION_1 { + global: + opendal_service_fs_bootstrap_v1; + local: + *; +}; diff --git a/bindings/python/dynamic-extensions/service-fs/src/lib.rs b/bindings/python/dynamic-extensions/service-fs/src/lib.rs new file mode 100644 index 000000000000..4d0da9d9dbae --- /dev/null +++ b/bindings/python/dynamic-extensions/service-fs/src/lib.rs @@ -0,0 +1,218 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::mem::size_of; +use std::panic::{AssertUnwindSafe, catch_unwind}; +use std::slice; + +use opendal_core::Operator; +use opendal_dynamic_extension_sdk::{ + ByteSlice, ExtensionApiV1, KeyValue, OutputBuffer, RUNTIME_PROTOCOL, STATUS_INVALID_ARGUMENT, + STATUS_OK, STATUS_OPERATION_FAILED, write_output, +}; +use opendal_service_fs::Fs; + +static API: ExtensionApiV1 = ExtensionApiV1 { + struct_size: size_of::(), + required_runtime_protocol: RUNTIME_PROTOCOL, + opendal_version: ByteSlice::from_static(opendal_core::raw::VERSION), + package_id: ByteSlice::from_static("opendal-service-fs-poc"), + component_id: ByteSlice::from_static("fs"), + entry_symbol: ByteSlice::from_static("opendal_service_fs_bootstrap_v1"), + create_operator, + destroy_operator, + operator_info, + operator_write, + operator_read, +}; + +static TOKIO_RUNTIME: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("prototype FS runtime should build") + }); + +fn fail(error: *mut OutputBuffer, message: impl AsRef) -> i32 { + let _ = unsafe { write_output(error, message.as_ref().as_bytes()) }; + STATUS_OPERATION_FAILED +} + +unsafe fn option_pairs<'a>( + options: *const KeyValue, + options_len: usize, +) -> Result, &'static str> { + if options_len == 0 { + return Ok(Vec::new()); + } + if options.is_null() { + return Err("options pointer is null"); + } + unsafe { slice::from_raw_parts(options, options_len) } + .iter() + .map(|pair| unsafe { Ok((pair.key.as_str()?, pair.value.as_str()?)) }) + .collect() +} + +unsafe extern "C" fn create_operator( + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if operator.is_null() { + return STATUS_INVALID_ARGUMENT; + } + *operator = std::ptr::null_mut(); + let options = match option_pairs(options, options_len) { + Ok(options) => options, + Err(message) => return fail(error, message), + }; + + let mut builder = Fs::default(); + for (key, value) in options { + builder = match key { + "root" => builder.root(value), + "atomic_write_dir" => builder.atomic_write_dir(value), + _ => return fail(error, format!("unsupported FS option {key}")), + }; + } + + match Operator::new(builder) { + Ok(value) => { + *operator = Box::into_raw(Box::new(value)).cast::(); + STATUS_OK + } + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "FS operator construction panicked")) +} + +unsafe extern "C" fn destroy_operator(operator: *mut c_void) { + if operator.is_null() { + return; + } + let _ = catch_unwind(AssertUnwindSafe(|| unsafe { + drop(Box::from_raw(operator.cast::())); + })); +} + +unsafe fn as_operator<'a>(operator: *mut c_void) -> Result<&'a Operator, &'static str> { + if operator.is_null() { + return Err("FS operator is null"); + } + Ok(unsafe { &*operator.cast::() }) +} + +unsafe extern "C" fn operator_info( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let info = operator.info(); + let value = serde_json::json!({ + "scheme": info.scheme().to_string(), + "name": info.name(), + "root": info.root(), + }); + write_output(output, value.to_string().as_bytes()) + })) + .unwrap_or_else(|_| fail(error, "FS operator info panicked")) +} + +unsafe extern "C" fn operator_write( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let data = match data.as_bytes() { + Ok(data) => data, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.write(path, data.to_vec()) { + Ok(_) => STATUS_OK, + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "FS write panicked")) +} + +unsafe extern "C" fn operator_read( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.read(path) { + Ok(value) => write_output(output, &value.to_vec()), + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "FS read panicked")) +} + +#[unsafe(no_mangle)] +/// Returns the FS extension function table. +/// +/// # Safety +/// +/// `extension` must point to writable storage for an extension table pointer. +pub unsafe extern "C" fn opendal_service_fs_bootstrap_v1( + extension: *mut *const ExtensionApiV1, +) -> i32 { + if extension.is_null() { + return STATUS_INVALID_ARGUMENT; + } + unsafe { *extension = &API }; + STATUS_OK +} diff --git a/bindings/python/dynamic-extensions/service-s3/Cargo.toml b/bindings/python/dynamic-extensions/service-s3/Cargo.toml new file mode 100644 index 000000000000..0c6a97e4e10b --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +edition.workspace = true +license.workspace = true +name = "s3-extension" +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] + +[dependencies] +opendal-core = { workspace = true, features = ["blocking"] } +opendal-dynamic-extension-sdk.workspace = true +opendal-service-s3.workspace = true +serde_json = "1" +tokio.workspace = true diff --git a/bindings/python/dynamic-extensions/service-s3/build.rs b/bindings/python/dynamic-extensions/service-s3/build.rs new file mode 100644 index 000000000000..501b9f44298a --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/build.rs @@ -0,0 +1,29 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() { + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } +} diff --git a/bindings/python/dynamic-extensions/service-s3/exports.map b/bindings/python/dynamic-extensions/service-s3/exports.map new file mode 100644 index 000000000000..09035fa37b48 --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_S3_EXTENSION_1 { + global: + opendal_service_s3_bootstrap_v1; + local: + *; +}; diff --git a/bindings/python/dynamic-extensions/service-s3/src/lib.rs b/bindings/python/dynamic-extensions/service-s3/src/lib.rs new file mode 100644 index 000000000000..7b904c14febe --- /dev/null +++ b/bindings/python/dynamic-extensions/service-s3/src/lib.rs @@ -0,0 +1,222 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::ffi::c_void; +use std::mem::size_of; +use std::panic::{AssertUnwindSafe, catch_unwind}; +use std::slice; + +use opendal_core::Operator; +use opendal_dynamic_extension_sdk::{ + ByteSlice, ExtensionApiV1, KeyValue, OutputBuffer, RUNTIME_PROTOCOL, STATUS_INVALID_ARGUMENT, + STATUS_OK, STATUS_OPERATION_FAILED, write_output, +}; +use opendal_service_s3::S3; + +static API: ExtensionApiV1 = ExtensionApiV1 { + struct_size: size_of::(), + required_runtime_protocol: RUNTIME_PROTOCOL, + opendal_version: ByteSlice::from_static(opendal_core::raw::VERSION), + package_id: ByteSlice::from_static("opendal-service-s3-poc"), + component_id: ByteSlice::from_static("s3"), + entry_symbol: ByteSlice::from_static("opendal_service_s3_bootstrap_v1"), + create_operator, + destroy_operator, + operator_info, + operator_write, + operator_read, +}; + +static TOKIO_RUNTIME: std::sync::LazyLock = + std::sync::LazyLock::new(|| { + tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("prototype S3 runtime should build") + }); + +fn fail(error: *mut OutputBuffer, message: impl AsRef) -> i32 { + let _ = unsafe { write_output(error, message.as_ref().as_bytes()) }; + STATUS_OPERATION_FAILED +} + +unsafe fn option_pairs<'a>( + options: *const KeyValue, + options_len: usize, +) -> Result, &'static str> { + if options_len == 0 { + return Ok(Vec::new()); + } + if options.is_null() { + return Err("options pointer is null"); + } + unsafe { slice::from_raw_parts(options, options_len) } + .iter() + .map(|pair| unsafe { Ok((pair.key.as_str()?, pair.value.as_str()?)) }) + .collect() +} + +unsafe extern "C" fn create_operator( + options: *const KeyValue, + options_len: usize, + operator: *mut *mut c_void, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + if operator.is_null() { + return STATUS_INVALID_ARGUMENT; + } + *operator = std::ptr::null_mut(); + let options = match option_pairs(options, options_len) { + Ok(options) => options, + Err(message) => return fail(error, message), + }; + + let mut builder = S3::default(); + for (key, value) in options { + builder = match key { + "bucket" => builder.bucket(value), + "region" => builder.region(value), + "root" => builder.root(value), + "endpoint" => builder.endpoint(value), + "access_key_id" => builder.access_key_id(value), + "secret_access_key" => builder.secret_access_key(value), + _ => return fail(error, format!("unsupported S3 option {key}")), + }; + } + + match Operator::new(builder) { + Ok(value) => { + *operator = Box::into_raw(Box::new(value)).cast::(); + STATUS_OK + } + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "S3 operator construction panicked")) +} + +unsafe extern "C" fn destroy_operator(operator: *mut c_void) { + if operator.is_null() { + return; + } + let _ = catch_unwind(AssertUnwindSafe(|| unsafe { + drop(Box::from_raw(operator.cast::())); + })); +} + +unsafe fn as_operator<'a>(operator: *mut c_void) -> Result<&'a Operator, &'static str> { + if operator.is_null() { + return Err("S3 operator is null"); + } + Ok(unsafe { &*operator.cast::() }) +} + +unsafe extern "C" fn operator_info( + operator: *mut c_void, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let info = operator.info(); + let value = serde_json::json!({ + "scheme": info.scheme().to_string(), + "name": info.name(), + "root": info.root(), + }); + write_output(output, value.to_string().as_bytes()) + })) + .unwrap_or_else(|_| fail(error, "S3 operator info panicked")) +} + +unsafe extern "C" fn operator_write( + operator: *mut c_void, + path: ByteSlice, + data: ByteSlice, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let data = match data.as_bytes() { + Ok(data) => data, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.write(path, data.to_vec()) { + Ok(_) => STATUS_OK, + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "S3 write panicked")) +} + +unsafe extern "C" fn operator_read( + operator: *mut c_void, + path: ByteSlice, + output: *mut OutputBuffer, + error: *mut OutputBuffer, +) -> i32 { + catch_unwind(AssertUnwindSafe(|| unsafe { + let operator = match as_operator(operator) { + Ok(operator) => operator, + Err(message) => return fail(error, message), + }; + let path = match path.as_str() { + Ok(path) => path, + Err(message) => return fail(error, message), + }; + let _guard = TOKIO_RUNTIME.enter(); + let blocking = match opendal_core::blocking::Operator::new(operator.clone()) { + Ok(blocking) => blocking, + Err(err) => return fail(error, err.to_string()), + }; + match blocking.read(path) { + Ok(value) => write_output(output, &value.to_vec()), + Err(err) => fail(error, err.to_string()), + } + })) + .unwrap_or_else(|_| fail(error, "S3 read panicked")) +} + +#[unsafe(no_mangle)] +/// Returns the S3 extension function table. +/// +/// # Safety +/// +/// `extension` must point to writable storage for an extension table pointer. +pub unsafe extern "C" fn opendal_service_s3_bootstrap_v1( + extension: *mut *const ExtensionApiV1, +) -> i32 { + if extension.is_null() { + return STATUS_INVALID_ARGUMENT; + } + unsafe { *extension = &API }; + STATUS_OK +} From 30c5c008c03384a7e815f35655320ddccf0690a9 Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:36:12 +0800 Subject: [PATCH 7/8] feat(python): compare PyO3 extension adapters --- bindings/python/dynamic-extensions/README.md | 5 + .../pyo3-comparison/.gitignore | 16 + .../pyo3-comparison/Cargo.lock | 1133 +++++++++++++++++ .../pyo3-comparison/Cargo.toml | 41 + .../pyo3-comparison/NOTES.md | 46 + .../pyo3-comparison/README.md | 58 + .../pyo3-comparison/base/Cargo.toml | 30 + .../pyo3-comparison/base/src/lib.rs | 22 + .../pyo3-comparison/bridge/Cargo.toml | 28 + .../pyo3-comparison/bridge/src/lib.rs | 127 ++ .../bridge/src/runtime_layer.rs | 229 ++++ .../pyo3-comparison/compare.py | 175 +++ .../pyo3-comparison/fs-capsule/Cargo.toml | 32 + .../pyo3-comparison/fs-capsule/src/lib.rs | 40 + .../pyo3-comparison/fs-direct/Cargo.toml | 32 + .../pyo3-comparison/fs-direct/src/lib.rs | 38 + .../pyo3-comparison/mime-capsule/Cargo.toml | 31 + .../pyo3-comparison/mime-capsule/src/lib.rs | 44 + .../pyo3-comparison/mime-direct/Cargo.toml | 31 + .../pyo3-comparison/mime-direct/src/lib.rs | 34 + .../pyo3-comparison/run-linux.sh | 54 + 21 files changed, 2246 insertions(+) create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/.gitignore create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/Cargo.lock create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/NOTES.md create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/README.md create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/base/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/base/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/bridge/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/runtime_layer.rs create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/compare.py create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/src/lib.rs create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/Cargo.toml create mode 100644 bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/src/lib.rs create mode 100755 bindings/python/dynamic-extensions/pyo3-comparison/run-linux.sh diff --git a/bindings/python/dynamic-extensions/README.md b/bindings/python/dynamic-extensions/README.md index 37ad40caa03e..7f9329564d4f 100644 --- a/bindings/python/dynamic-extensions/README.md +++ b/bindings/python/dynamic-extensions/README.md @@ -21,6 +21,11 @@ This in-tree POC tests the package and loading model proposed in [`bindings/docs/dynamic-extensions/python.md`](../../docs/dynamic-extensions/python.md). It does not change the released Python binding. +The current-PyO3 [comparison prototype](pyo3-comparison/README.md) separately +tests an FS service and MIME layer with both capsule and direct PyO3 adapters. +This directory's original POC instead tests the wider versioned operation-table +boundary proposed by the design document. + The workspace builds three independently linked native artifacts: - `opendal-runtime-poc` owns protocol negotiation, service registration, diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/.gitignore b/bindings/python/dynamic-extensions/pyo3-comparison/.gitignore new file mode 100644 index 000000000000..d7d2adf88a85 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/.gitignore @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +/target/ diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.lock b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.lock new file mode 100644 index 000000000000..3044d0306635 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.lock @@ -0,0 +1,1133 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "anyhow" +version = "1.0.104" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470" + +[[package]] +name = "base64" +version = "0.23.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5" + +[[package]] +name = "bitflags" +version = "1.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "block-buffer" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + +[[package]] +name = "crypto-common" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" +dependencies = [ + "hybrid-array", +] + +[[package]] +name = "defmt" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e2953bfe4f93bbd20cc71198842756f77d161884c99ebbabc41d80231ded88d1" +dependencies = [ + "bitflags 1.3.2", + "defmt-macros", +] + +[[package]] +name = "defmt-macros" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bad9c72e7ca2137e0dc3813245a0d282fd6daad32fd800af018306a9169b5fe8" +dependencies = [ + "defmt-parser", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "defmt-parser" +version = "1.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "10d60334b3b2e7c9d91ef8150abfb6fa4c1c39ebbcf4a81c2e346aad939fee3e" +dependencies = [ + "thiserror", +] + +[[package]] +name = "digest" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" +dependencies = [ + "block-buffer", + "const-oid", + "crypto-common", +] + +[[package]] +name = "displaydoc" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "errno" +version = "0.3.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" +dependencies = [ + "libc", + "windows-sys", +] + +[[package]] +name = "form_urlencoded" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" +dependencies = [ + "percent-encoding", +] + +[[package]] +name = "futures" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a88cf1f829d945f548cf8fec32c61b1f202b6d93b45848602fc02af4b12ad218" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "262590f4fe6afeb0bc83be1daa64e52657fe185690a958af7f3ad0e92085c5ae" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2cd50c473c80f6d7c3670a752354b8e569b1a7cbfdc0419ec88e5edad85e0dc7" + +[[package]] +name = "futures-io" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4577ecaa3c4f96589d473f679a71b596316f6641bc350038b962a5daf0085d7a" + +[[package]] +name = "futures-macro" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d6d3cde68c518367be28956066ddfef33813991b77a55005a69dae04bf3b10b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "futures-sink" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e34418ac499d6305c2fb5ad0ed2f6ac998c5f8ca209b4510f7f94242c647e307" + +[[package]] +name = "futures-task" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b231ed28831efb4a61a08580c4bc233ec56bc009f4cd8f52da2c3cb97df0c109" + +[[package]] +name = "futures-util" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a77a90a256fce34da66415271e30f94ee91c57b04b8a2c042d9cf3220179deaa" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "getrandom" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" +dependencies = [ + "cfg-if", + "libc", + "r-efi", +] + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "http" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "918d3568bebf352712bc2ef3d46a8bcf1a75b373be6539de198e9105cbbf9ce0" +dependencies = [ + "bytes", + "itoa", +] + +[[package]] +name = "hybrid-array" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "707114b52a152fa7bdb290cd7cd5912d9467273b6d74e21b8d81aca1f8533f6b" +dependencies = [ + "typenum", +] + +[[package]] +name = "icu_collections" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" +dependencies = [ + "displaydoc", + "potential_utf", + "utf8_iter", + "yoke", + "zerofrom", + "zerovec", +] + +[[package]] +name = "icu_locale_core" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" +dependencies = [ + "displaydoc", + "litemap", + "tinystr", + "writeable", + "zerovec", +] + +[[package]] +name = "icu_normalizer" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" +dependencies = [ + "icu_collections", + "icu_normalizer_data", + "icu_properties", + "icu_provider", + "smallvec", + "zerovec", +] + +[[package]] +name = "icu_normalizer_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" + +[[package]] +name = "icu_properties" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" +dependencies = [ + "icu_collections", + "icu_locale_core", + "icu_properties_data", + "icu_provider", + "zerotrie", + "zerovec", +] + +[[package]] +name = "icu_properties_data" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" + +[[package]] +name = "icu_provider" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" +dependencies = [ + "displaydoc", + "icu_locale_core", + "writeable", + "yoke", + "zerofrom", + "zerotrie", + "zerovec", +] + +[[package]] +name = "idna" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" +dependencies = [ + "idna_adapter", + "smallvec", + "utf8_iter", +] + +[[package]] +name = "idna_adapter" +version = "1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" +dependencies = [ + "icu_normalizer", + "icu_properties", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jiff" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "668b7183bd07af9a4885f5c35b0cc5c83c4607a913c16b7e17291832910d2dcc" +dependencies = [ + "defmt", + "jiff-core", + "jiff-static", + "jiff-tzdb-platform", + "js-sys", + "log", + "portable-atomic", + "portable-atomic-util", + "serde_core", + "wasm-bindgen", + "windows-link", +] + +[[package]] +name = "jiff-core" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7feca88439efe53da3754500c1851dedf3cb36c524dd5cf8225cc0794de95d09" +dependencies = [ + "defmt", +] + +[[package]] +name = "jiff-static" +version = "0.2.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a69dcb3a21cfb32ce1cd056169337ca284af0766dd766e7878819b251a49204" +dependencies = [ + "jiff-core", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "jiff-tzdb" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "142bd39932ad231f10513df9ab62661fead8719872150b7ad02a2df79f4e141e" + +[[package]] +name = "jiff-tzdb-platform" +version = "0.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "875a5a69ac2bab1a891711cf5eccbec1ce0341ea805560dcd90b7a2e925132e8" +dependencies = [ + "jiff-tzdb", +] + +[[package]] +name = "js-sys" +version = "0.3.103" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53b44bfcdb3f8d5837a46dae1ca9660a837176eee74a28b229bc626816589102" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "linux-raw-sys" +version = "0.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" + +[[package]] +name = "litemap" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" + +[[package]] +name = "log" +version = "0.4.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" + +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest", +] + +[[package]] +name = "mea" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "31fc7d159de0085ab6dd7ff145a9819442cfd3d098f783263120503c3f3e58b0" +dependencies = [ + "slab", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "mime" +version = "0.3.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a" + +[[package]] +name = "mime_guess" +version = "2.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e" +dependencies = [ + "mime", + "unicase", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "opendal-core" +version = "0.58.1" +dependencies = [ + "anyhow", + "base64", + "bytes", + "futures", + "http", + "jiff", + "log", + "md-5", + "mea", + "percent-encoding", + "quick-xml", + "serde", + "serde_json", + "tokio", + "url", + "uuid", + "web-time", +] + +[[package]] +name = "opendal-layer-mime-guess" +version = "0.58.1" +dependencies = [ + "mime_guess", + "opendal-core", +] + +[[package]] +name = "opendal-service-fs" +version = "0.58.1" +dependencies = [ + "bytes", + "log", + "opendal-core", + "serde", + "tokio", + "xattr", +] + +[[package]] +name = "percent-encoding" +version = "2.3.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" + +[[package]] +name = "pin-project" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" +dependencies = [ + "pin-project-internal", +] + +[[package]] +name = "pin-project-internal" +version = "1.1.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "portable-atomic" +version = "1.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d20d5497ef88037a52ff98267d066e7f11fcc5e99bbfbd58a42336193aacec3" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "potential_utf" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" +dependencies = [ + "zerovec", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "pyo3" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cd274650b21d4bfc26a0a47587962c1edb425f69287324355cd040c3ea66071c" +dependencies = [ + "libc", + "once_cell", + "portable-atomic", + "pyo3-build-config", + "pyo3-ffi", + "pyo3-macros", +] + +[[package]] +name = "pyo3-build-config" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c5e2a7d2f0d013342f295c048ad19237add5154a55b1c5a254c0ec93d4109078" +dependencies = [ + "target-lexicon", +] + +[[package]] +name = "pyo3-comparison-base" +version = "0.0.0" +dependencies = [ + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-bridge" +version = "0.0.0" +dependencies = [ + "opendal-core", + "pin-project", + "pyo3", + "tokio", +] + +[[package]] +name = "pyo3-comparison-fs-capsule" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-service-fs", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-fs-direct" +version = "0.0.0" +dependencies = [ + "opendal-core", + "opendal-service-fs", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-mime-capsule" +version = "0.0.0" +dependencies = [ + "opendal-layer-mime-guess", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-comparison-mime-direct" +version = "0.0.0" +dependencies = [ + "opendal-layer-mime-guess", + "pyo3", + "pyo3-comparison-bridge", +] + +[[package]] +name = "pyo3-ffi" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ca85c467da1bbc8d866eea5deff9cf29ea5f7785054a17da36e65bda9c05845b" +dependencies = [ + "libc", + "pyo3-build-config", +] + +[[package]] +name = "pyo3-macros" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ac53762fd065daa3194dd09337a38bd793a188100fd1a9304c4ab312d901771" +dependencies = [ + "proc-macro2", + "pyo3-macros-backend", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "pyo3-macros-backend" +version = "0.29.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ca3a1557399783172dc5bf39cfca835157732532cba56b71d2292161e53b362" +dependencies = [ + "heck", + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "quick-xml" +version = "0.41.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1" +dependencies = [ + "memchr", + "serde", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rustix" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" +dependencies = [ + "bitflags 2.13.1", + "errno", + "libc", + "linux-raw-sys", + "windows-sys", +] + +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + +[[package]] +name = "serde" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4148590afebada386688f18773da617792bf2ef03ffc1e4cbd2b1d45b023e0ba" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "67dca2c9c51e58a4791a4b1ed58308b39c64224d349a935ab5039aa360942a48" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.229" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "serde_json" +version = "1.0.151" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c841b55ecdae098c80dcae9cf767f6f8a0c2cdb3416bbef72181df4d0fe73f14" +dependencies = [ + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "stable_deref_trait" +version = "1.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "synstructure" +version = "0.13.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "target-lexicon" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" + +[[package]] +name = "thiserror" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09a43598840e33d5b0331f38c5e30d13bb11c11210a4b58f0d9b18a5a5eefcd9" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "43cbfe0cf76104d42a574802844187e84a305e531ed54455f11fbde0f10541cd" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "tinystr" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" +dependencies = [ + "displaydoc", + "zerovec", +] + +[[package]] +name = "tokio" +version = "1.53.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "202caea871b69668250d242070849eb495be178ed697a3e98aebce5bc81a0bed" +dependencies = [ + "bytes", + "pin-project-lite", + "tokio-macros", +] + +[[package]] +name = "tokio-macros" +version = "2.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.3", +] + +[[package]] +name = "typenum" +version = "1.20.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" + +[[package]] +name = "unicase" +version = "2.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "url" +version = "2.5.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" +dependencies = [ + "form_urlencoded", + "idna", + "percent-encoding", + "serde", +] + +[[package]] +name = "utf8_iter" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" + +[[package]] +name = "uuid" +version = "1.24.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239" +dependencies = [ + "getrandom", + "js-sys", + "serde_core", + "wasm-bindgen", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4b067c0c11094aef6b7a801c1e34a26affafdf3d051dba08456b868789aaf9a4" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "167ce5e579f6bcf889c4f7175a8a5a585de84e8ff93976ce393efa5f2837aab1" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f3997c7839262f4ef12cf90b818d6340c18e80f263f1a94bf157d0ec4420380e" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn 2.0.119", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.126" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc1b4cb0cc549fcf58d7dfc081778139b3d283a081644e833e84682ad71cea24" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "web-time" +version = "1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" +dependencies = [ + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "writeable" +version = "0.6.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" + +[[package]] +name = "xattr" +version = "1.6.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "32e45ad4206f6d2479085147f02bc2ef834ac85886624a23575ae137c8aa8156" +dependencies = [ + "libc", + "rustix", +] + +[[package]] +name = "yoke" +version = "0.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" +dependencies = [ + "stable_deref_trait", + "yoke-derive", + "zerofrom", +] + +[[package]] +name = "yoke-derive" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerofrom" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" +dependencies = [ + "zerofrom-derive", +] + +[[package]] +name = "zerofrom-derive" +version = "0.1.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", + "synstructure", +] + +[[package]] +name = "zerotrie" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" +dependencies = [ + "displaydoc", + "yoke", + "zerofrom", +] + +[[package]] +name = "zerovec" +version = "0.11.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" +dependencies = [ + "yoke", + "zerofrom", + "zerovec-derive", +] + +[[package]] +name = "zerovec-derive" +version = "0.11.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + +[[package]] +name = "zmij" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29666d0abbfad1e3dc4dcf6144730dd3a3ab225bbbdac83319345b1b44ccfc1b" diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.toml new file mode 100644 index 000000000000..be0abb7a0bb8 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/Cargo.toml @@ -0,0 +1,41 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +members = [ + "base", + "bridge", + "fs-capsule", + "fs-direct", + "mime-capsule", + "mime-direct", +] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" + +[workspace.dependencies] +opendal-core = { path = "../../../../core/core", features = ["blocking"] } +opendal-layer-mime-guess = { path = "../../../../core/layers/mime-guess" } +opendal-service-fs = { path = "../../../../core/services/fs" } +pin-project = "1" +pyo3 = { version = "0.29.0", features = ["extension-module"] } +pyo3-comparison-bridge = { path = "bridge" } +tokio = { version = "1", features = ["fs", "rt-multi-thread"] } diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/NOTES.md b/bindings/python/dynamic-extensions/pyo3-comparison/NOTES.md new file mode 100644 index 000000000000..8bbee9c2525e --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/NOTES.md @@ -0,0 +1,46 @@ + + +# Prototype Notes + +Question: Can separately built FS and MIME layer packages return and transform +the base package's PyO3 `Operator` with and without capsules? + +Observed result: + +- The capsule FS factory returns the exact base-package Python `Operator` type. +- FS write, read, and `stat` succeed after the operator crosses into the base + package, proving that the service-local runtime layer and executor enter the + FS package's Tokio context for these operations. +- The capsule MIME package accepts that operator and changes `hello.txt` from no + content type to `text/plain`. +- The direct FS package returns an object displayed as + `opendal_poc.Operator`, but its Python type object differs from the base + package's type object. +- The direct MIME package rejects both the FS package's operator and the base + package's operator with `Operator object is not an instance of Operator`. + +Conclusion: ordinary Rust dependency reuse does not share a PyO3 class between +independently linked extensions. A capsule can preserve the base Python class +interface, but this prototype's capsule payload is still a Rust `Operator` and +therefore requires an exact build. A production capsule should carry a +versioned function table or another explicitly validated ownership contract +instead of treating the Rust layout as stable. + +Known limit: the runtime-switch layer covers service methods, readers, writers, +and the executor used by this experiment. A production implementation must +also wrap returned listers, deleters, and copiers. diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/README.md b/bindings/python/dynamic-extensions/pyo3-comparison/README.md new file mode 100644 index 000000000000..02899e9c4361 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/README.md @@ -0,0 +1,58 @@ + + +# PyO3 Extension Comparison Prototype + +PROTOTYPE: delete or absorb this directory after the design question is +answered. + +This experiment asks whether independently built FS and MIME layer packages can +preserve the base package's PyO3 `Operator` interface. It compares two adapters: + +- The capsule adapter passes an exact-build `opendal_core::Operator` through + named `PyCapsule` values. +- The direct adapter reuses the Rust `PyOperator` definition through ordinary + Cargo dependencies and lets PyO3 perform extraction. + +Both adapters compile from the same sources in separate target directories. +The FS package injects a service-local Tokio runtime layer. The MIME package +applies `MimeGuessLayer`, whose effect is visible through +`Operator.content_type("hello.txt")`. + +Run both paths: + +```console +./run-linux.sh +``` + +Run the interactive state viewer: + +```console +./run-linux.sh --interactive +``` + +The capsule is an unsafe exact-build experiment, not a stable ABI. Its payload +contains a Rust `Operator`, so compiler, dependency graph, flags, and OpenDAL +source must match even though the capsule name is versioned. + +This comparison adapts the capsule delegation and runtime-switch patterns from +the earlier [split Python binding prototype][split-prototype]. It directly +tests the independently linked `#[pyclass]` constraint discussed in +[PyO3 issue #1444][pyo3-1444] against the current OpenDAL core APIs. + +[pyo3-1444]: https://github.com/PyO3/pyo3/issues/1444 +[split-prototype]: https://github.com/chitralverma/opendal-python-bindings/blob/main/pyo3-opendal/src/layers.rs diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/base/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/base/Cargo.toml new file mode 100644 index 000000000000..15221e2f5eb5 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/base/Cargo.toml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-base" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "base" + +[dependencies] +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/base/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/base/src/lib.rs new file mode 100644 index 000000000000..9e94095b14a2 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/base/src/lib.rs @@ -0,0 +1,22 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use pyo3::prelude::*; +use pyo3_comparison_bridge::PyOperator; + +#[pymodule(gil_used = false)] +fn opendal_poc(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::() +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/bridge/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/Cargo.toml new file mode 100644 index 000000000000..9c9c144649ff --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/Cargo.toml @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-bridge" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[dependencies] +opendal-core.workspace = true +pin-project.workspace = true +pyo3.workspace = true +tokio.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/lib.rs new file mode 100644 index 000000000000..27a1fc340105 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/lib.rs @@ -0,0 +1,127 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +mod runtime_layer; + +use std::ffi::CStr; +use std::sync::LazyLock; + +use opendal_core::Operator; +use pyo3::intern; +use pyo3::prelude::*; +use pyo3::types::{PyBytes, PyCapsule, PyCapsuleMethods}; + +pub use runtime_layer::RuntimeLayer; + +const OPERATOR_CAPSULE_NAME: &CStr = c"opendal.poc.operator.v1"; + +pub fn runtime() -> &'static tokio::runtime::Runtime { + static RUNTIME: LazyLock = LazyLock::new(|| { + tokio::runtime::Builder::new_multi_thread() + .enable_all() + .build() + .expect("prototype Tokio runtime should build") + }); + &RUNTIME +} + +pub fn format_error(error: opendal_core::Error) -> PyErr { + pyo3::exceptions::PyRuntimeError::new_err(error.to_string()) +} + +pub fn to_operator_capsule(py: Python<'_>, op: Operator) -> PyResult> { + PyCapsule::new_with_value(py, op, OPERATOR_CAPSULE_NAME) +} + +pub fn from_operator_capsule(capsule: &Bound<'_, PyCapsule>) -> PyResult { + let pointer = capsule + .pointer_checked(Some(OPERATOR_CAPSULE_NAME))? + .cast::(); + Ok(unsafe { pointer.as_ref().clone() }) +} + +#[pyclass(module = "opendal_poc", name = "Operator")] +pub struct PyOperator { + op: Operator, +} + +impl PyOperator { + pub fn from_async(op: Operator) -> Self { + Self { op } + } + + pub fn async_operator(&self) -> &Operator { + &self.op + } +} + +#[pymethods] +impl PyOperator { + #[new] + fn new() -> PyResult { + let _guard = runtime().enter(); + let op = Operator::new(opendal_core::services::Memory::default()).map_err(format_error)?; + Ok(Self { op }) + } + + #[staticmethod] + fn _from_capsule(capsule: &Bound<'_, PyCapsule>) -> PyResult { + Ok(Self { + op: from_operator_capsule(capsule)?, + }) + } + + fn layer(&self, py: Python<'_>, layer: &Bound<'_, PyAny>) -> PyResult { + let capsule = to_operator_capsule(py, self.op.clone())?; + let result = layer.call_method1(intern!(py, "_layer_apply"), (capsule,))?; + let result = result.cast::()?; + Self::_from_capsule(result) + } + + fn scheme(&self) -> String { + self.op.info().scheme().to_string() + } + + fn content_type(&self, py: Python<'_>, path: String) -> PyResult> { + let op = self.op.clone(); + py.detach(move || { + runtime() + .block_on(op.stat(&path)) + .map(|metadata| metadata.content_type().map(str::to_string)) + .map_err(format_error) + }) + } + + fn write(&self, py: Python<'_>, path: String, content: Vec) -> PyResult<()> { + let op = self.op.clone(); + py.detach(move || { + runtime() + .block_on(op.write(&path, content)) + .map(|_| ()) + .map_err(format_error) + }) + } + + fn read<'py>(&self, py: Python<'py>, path: String) -> PyResult> { + let op = self.op.clone(); + let content = py.detach(move || { + runtime() + .block_on(op.read(&path)) + .map(|buffer| buffer.to_vec()) + .map_err(format_error) + })?; + Ok(PyBytes::new(py, &content)) + } +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/runtime_layer.rs b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/runtime_layer.rs new file mode 100644 index 000000000000..323167b01466 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/bridge/src/runtime_layer.rs @@ -0,0 +1,229 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::future::Future; +use std::sync::Arc; +use std::task::{Context, Poll}; + +use opendal_core::raw::*; +use opendal_core::*; +use pin_project::pin_project; + +#[derive(Clone, Debug)] +pub struct RuntimeLayer { + handle: tokio::runtime::Handle, +} + +impl RuntimeLayer { + pub fn new(handle: tokio::runtime::Handle) -> Self { + Self { handle } + } +} + +impl Layer for RuntimeLayer { + fn apply_service(&self, inner: Servicer) -> Servicer { + Arc::new(RuntimeService { + inner, + handle: self.handle.clone(), + }) + } + + fn apply_context(&self, _service: Servicer, inner: OperationContext) -> OperationContext { + inner.with_executor(Executor::with(RuntimeExecutor { + handle: self.handle.clone(), + })) + } +} + +#[derive(Clone)] +struct RuntimeExecutor { + handle: tokio::runtime::Handle, +} + +impl Execute for RuntimeExecutor { + fn execute(&self, future: BoxedStaticFuture<()>) { + drop(self.handle.spawn(future)); + } +} + +#[derive(Debug)] +struct RuntimeService { + inner: Servicer, + handle: tokio::runtime::Handle, +} + +impl Service for RuntimeService { + type Reader = RuntimeReader; + type Writer = RuntimeWriter; + type Lister = oio::Lister; + type Deleter = oio::Deleter; + type Copier = oio::Copier; + + fn info(&self) -> ServiceInfo { + self.inner.info() + } + + fn capability(&self) -> Capability { + self.inner.capability() + } + + async fn create_dir( + &self, + ctx: &OperationContext, + path: &str, + args: OpCreateDir, + ) -> Result { + RuntimeFuture::new(self.inner.create_dir(ctx, path, args), self.handle.clone()).await + } + + async fn stat(&self, ctx: &OperationContext, path: &str, args: OpStat) -> Result { + RuntimeFuture::new(self.inner.stat(ctx, path, args), self.handle.clone()).await + } + + fn read(&self, ctx: &OperationContext, path: &str, args: OpRead) -> Result { + let _guard = self.handle.enter(); + self.inner.read(ctx, path, args).map(|inner| RuntimeReader { + inner, + handle: self.handle.clone(), + }) + } + + fn write(&self, ctx: &OperationContext, path: &str, args: OpWrite) -> Result { + let _guard = self.handle.enter(); + self.inner + .write(ctx, path, args) + .map(|inner| RuntimeWriter { + inner, + handle: self.handle.clone(), + }) + } + + fn delete(&self, ctx: &OperationContext) -> Result { + let _guard = self.handle.enter(); + self.inner.delete(ctx) + } + + fn list(&self, ctx: &OperationContext, path: &str, args: OpList) -> Result { + let _guard = self.handle.enter(); + self.inner.list(ctx, path, args) + } + + fn copy( + &self, + ctx: &OperationContext, + from: &str, + to: &str, + args: OpCopy, + opts: OpCopier, + ) -> Result { + let _guard = self.handle.enter(); + self.inner.copy(ctx, from, to, args, opts) + } + + async fn rename( + &self, + ctx: &OperationContext, + from: &str, + to: &str, + args: OpRename, + ) -> Result { + RuntimeFuture::new(self.inner.rename(ctx, from, to, args), self.handle.clone()).await + } + + async fn presign( + &self, + ctx: &OperationContext, + path: &str, + args: OpPresign, + ) -> Result { + RuntimeFuture::new(self.inner.presign(ctx, path, args), self.handle.clone()).await + } +} + +struct RuntimeReader { + inner: oio::Reader, + handle: tokio::runtime::Handle, +} + +impl oio::Read for RuntimeReader { + async fn open(&self, range: BytesRange) -> Result<(RpRead, Box)> { + let (response, stream) = + RuntimeFuture::new(self.inner.open(range), self.handle.clone()).await?; + Ok(( + response, + Box::new(RuntimeReadStream { + inner: stream, + handle: self.handle.clone(), + }), + )) + } + + async fn read(&self, range: BytesRange) -> Result<(RpRead, Buffer)> { + RuntimeFuture::new(self.inner.read(range), self.handle.clone()).await + } +} + +struct RuntimeReadStream { + inner: Box, + handle: tokio::runtime::Handle, +} + +impl oio::ReadStream for RuntimeReadStream { + async fn read(&mut self) -> Result { + RuntimeFuture::new(self.inner.read(), self.handle.clone()).await + } +} + +struct RuntimeWriter { + inner: oio::Writer, + handle: tokio::runtime::Handle, +} + +impl oio::Write for RuntimeWriter { + async fn write(&mut self, buffer: Buffer) -> Result<()> { + RuntimeFuture::new(self.inner.write(buffer), self.handle.clone()).await + } + + async fn close(&mut self) -> Result { + RuntimeFuture::new(self.inner.close(), self.handle.clone()).await + } + + async fn abort(&mut self) -> Result<()> { + RuntimeFuture::new(self.inner.abort(), self.handle.clone()).await + } +} + +#[pin_project] +struct RuntimeFuture { + #[pin] + inner: F, + handle: tokio::runtime::Handle, +} + +impl RuntimeFuture { + fn new(inner: F, handle: tokio::runtime::Handle) -> Self { + Self { inner, handle } + } +} + +impl Future for RuntimeFuture { + type Output = F::Output; + + fn poll(self: std::pin::Pin<&mut Self>, context: &mut Context<'_>) -> Poll { + let this = self.project(); + let _guard = this.handle.enter(); + this.inner.poll(context) + } +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/compare.py b/bindings/python/dynamic-extensions/pyo3-comparison/compare.py new file mode 100644 index 000000000000..747ceee02feb --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/compare.py @@ -0,0 +1,175 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from __future__ import annotations + +import argparse +import json +import tempfile +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Callable + +import opendal_fs_capsule +import opendal_fs_direct +import opendal_mime_capsule +import opendal_mime_direct +import opendal_poc + + +@dataclass +class VariantState: + attempted: bool = False + operator_type: str | None = None + operator_type_identity: int | None = None + is_base_operator: bool | None = None + fs_round_trip: bool | None = None + fs_stat_succeeded: bool | None = None + fs_stat_before_layer: str | None = None + fs_stat_after_layer: str | None = None + layer_applied: bool | None = None + layer_error: str | None = None + base_operator_layer_error: str | None = None + error: str | None = None + + +@dataclass +class ComparisonState: + base_operator_type_identity: int + base_memory_scheme: str + capsule: VariantState + direct: VariantState + + +def initial_state() -> ComparisonState: + return ComparisonState( + base_operator_type_identity=id(opendal_poc.Operator), + base_memory_scheme=opendal_poc.Operator().scheme(), + capsule=VariantState(), + direct=VariantState(), + ) + + +def run_capsule(root: Path) -> VariantState: + state = VariantState(attempted=True) + try: + variant_root = root / "capsule" + variant_root.mkdir(exist_ok=True) + operator = opendal_fs_capsule.create(str(variant_root)) + state.operator_type = repr(type(operator)) + state.operator_type_identity = id(type(operator)) + state.is_base_operator = isinstance(operator, opendal_poc.Operator) + operator.write("hello.txt", b"hello from capsule") + state.fs_round_trip = operator.read("hello.txt") == b"hello from capsule" + state.fs_stat_before_layer = operator.content_type("hello.txt") + state.fs_stat_succeeded = True + layered = operator.layer(opendal_mime_capsule.MimeGuessLayer()) + state.fs_stat_after_layer = layered.content_type("hello.txt") + state.layer_applied = state.fs_stat_after_layer == "text/plain" + except Exception as error: + state.error = f"{type(error).__name__}: {error}" + return state + + +def run_direct(root: Path) -> VariantState: + state = VariantState(attempted=True) + try: + variant_root = root / "direct" + variant_root.mkdir(exist_ok=True) + operator = opendal_fs_direct.create(str(variant_root)) + state.operator_type = repr(type(operator)) + state.operator_type_identity = id(type(operator)) + state.is_base_operator = isinstance(operator, opendal_poc.Operator) + operator.write("hello.txt", b"hello from direct") + state.fs_round_trip = operator.read("hello.txt") == b"hello from direct" + state.fs_stat_before_layer = operator.content_type("hello.txt") + state.fs_stat_succeeded = True + try: + layered = opendal_mime_direct.apply(operator) + except Exception as error: + state.layer_error = f"{type(error).__name__}: {error}" + state.layer_applied = False + else: + state.fs_stat_after_layer = layered.content_type("hello.txt") + state.layer_applied = state.fs_stat_after_layer == "text/plain" + + try: + opendal_mime_direct.apply(opendal_poc.Operator()) + except Exception as error: + state.base_operator_layer_error = f"{type(error).__name__}: {error}" + except Exception as error: + state.error = f"{type(error).__name__}: {error}" + return state + + +def render(state: ComparisonState, *, clear: bool) -> None: + if clear: + print("\033[2J\033[H", end="") + print("\033[1mPyO3 extension comparison\033[0m") + print(json.dumps(asdict(state), indent=2, sort_keys=True)) + print() + print( + "\033[1m[c]\033[0m capsule " + "\033[1m[d]\033[0m direct " + "\033[1m[a]\033[0m both " + "\033[1m[r]\033[0m reset " + "\033[1m[q]\033[0m quit" + ) + + +def apply_action( + state: ComparisonState, + action: str, + root: Path, +) -> ComparisonState: + actions: dict[str, Callable[[Path], VariantState]] = { + "c": run_capsule, + "d": run_direct, + } + if action == "r": + return initial_state() + if action == "a": + state.capsule = run_capsule(root) + state.direct = run_direct(root) + elif action in actions: + setattr(state, "capsule" if action == "c" else "direct", actions[action](root)) + return state + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--all", action="store_true") + parser.add_argument("--interactive", action="store_true") + args = parser.parse_args() + + with tempfile.TemporaryDirectory(prefix="opendal-pyo3-comparison-") as root: + root_path = Path(root) + state = initial_state() + + if args.all or not args.interactive: + state = apply_action(state, "a", root_path) + render(state, clear=False) + return + + while True: + render(state, clear=True) + action = input("> ").strip().lower()[:1] + if action == "q": + return + state = apply_action(state, action, root_path) + + +if __name__ == "__main__": + main() diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/Cargo.toml new file mode 100644 index 000000000000..6bc6447fa3b4 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-fs-capsule" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "fs_capsule" + +[dependencies] +opendal-core.workspace = true +opendal-service-fs.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/src/lib.rs new file mode 100644 index 000000000000..b43cbd242165 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-capsule/src/lib.rs @@ -0,0 +1,40 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_core::Operator; +use opendal_service_fs::Fs; +use pyo3::prelude::*; +use pyo3_comparison_bridge::{RuntimeLayer, format_error, runtime, to_operator_capsule}; + +fn build_fs(root: &str) -> PyResult { + let local_runtime = runtime(); + let _guard = local_runtime.enter(); + Operator::new(Fs::default().root(root)) + .map(|operator| operator.layer(RuntimeLayer::new(local_runtime.handle().clone()))) + .map_err(format_error) +} + +#[pyfunction] +fn create<'py>(py: Python<'py>, root: &str) -> PyResult> { + let capsule = to_operator_capsule(py, build_fs(root)?)?; + py.import("opendal_poc")? + .getattr("Operator")? + .call_method1("_from_capsule", (capsule,)) +} + +#[pymodule(gil_used = false)] +fn opendal_fs_capsule(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_function(wrap_pyfunction!(create, module)?) +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/Cargo.toml new file mode 100644 index 000000000000..3b3129cc39a2 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/Cargo.toml @@ -0,0 +1,32 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-fs-direct" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "fs_direct" + +[dependencies] +opendal-core.workspace = true +opendal-service-fs.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/src/lib.rs new file mode 100644 index 000000000000..4bd319f28cce --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/fs-direct/src/lib.rs @@ -0,0 +1,38 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_core::Operator; +use opendal_service_fs::Fs; +use pyo3::prelude::*; +use pyo3_comparison_bridge::{PyOperator, RuntimeLayer, format_error, runtime}; + +fn build_fs(root: &str) -> PyResult { + let local_runtime = runtime(); + let _guard = local_runtime.enter(); + Operator::new(Fs::default().root(root)) + .map(|operator| operator.layer(RuntimeLayer::new(local_runtime.handle().clone()))) + .map_err(format_error) +} + +#[pyfunction] +fn create(root: &str) -> PyResult { + build_fs(root).map(PyOperator::from_async) +} + +#[pymodule(gil_used = false)] +fn opendal_fs_direct(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::()?; + module.add_function(wrap_pyfunction!(create, module)?) +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/Cargo.toml new file mode 100644 index 000000000000..0c60d6c03fa5 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/Cargo.toml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-mime-capsule" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "mime_capsule" + +[dependencies] +opendal-layer-mime-guess.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/src/lib.rs new file mode 100644 index 000000000000..fde68a665a31 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-capsule/src/lib.rs @@ -0,0 +1,44 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_layer_mime_guess::MimeGuessLayer as CoreMimeGuessLayer; +use pyo3::prelude::*; +use pyo3::types::PyCapsule; +use pyo3_comparison_bridge::{from_operator_capsule, to_operator_capsule}; + +#[pyclass(name = "MimeGuessLayer")] +struct MimeGuessLayer; + +#[pymethods] +impl MimeGuessLayer { + #[new] + fn new() -> Self { + Self + } + + fn _layer_apply<'py>( + &self, + py: Python<'py>, + capsule: &Bound<'py, PyCapsule>, + ) -> PyResult> { + let operator = from_operator_capsule(capsule)?; + to_operator_capsule(py, operator.layer(CoreMimeGuessLayer::default())) + } +} + +#[pymodule(gil_used = false)] +fn opendal_mime_capsule(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::() +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/Cargo.toml b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/Cargo.toml new file mode 100644 index 000000000000..9d562fd5f542 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/Cargo.toml @@ -0,0 +1,31 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "pyo3-comparison-mime-direct" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "mime_direct" + +[dependencies] +opendal-layer-mime-guess.workspace = true +pyo3.workspace = true +pyo3-comparison-bridge.workspace = true diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/src/lib.rs b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/src/lib.rs new file mode 100644 index 000000000000..b1f37eb19c79 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/mime-direct/src/lib.rs @@ -0,0 +1,34 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use opendal_layer_mime_guess::MimeGuessLayer; +use pyo3::prelude::*; +use pyo3_comparison_bridge::PyOperator; + +#[pyfunction] +fn apply(operator: PyRef<'_, PyOperator>) -> PyOperator { + PyOperator::from_async( + operator + .async_operator() + .clone() + .layer(MimeGuessLayer::default()), + ) +} + +#[pymodule(gil_used = false)] +fn opendal_mime_direct(module: &Bound<'_, PyModule>) -> PyResult<()> { + module.add_class::()?; + module.add_function(wrap_pyfunction!(apply, module)?) +} diff --git a/bindings/python/dynamic-extensions/pyo3-comparison/run-linux.sh b/bindings/python/dynamic-extensions/pyo3-comparison/run-linux.sh new file mode 100755 index 000000000000..0d867c2613b1 --- /dev/null +++ b/bindings/python/dynamic-extensions/pyo3-comparison/run-linux.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +prototype_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +target_root="$prototype_dir/target" +stage="$target_root/python-stage" +audit="$prototype_dir/../audit-elf-exports.sh" + +build_package() { + package=$1 + target=$2 + CARGO_TARGET_DIR="$target_root/$target" cargo build --release --locked --offline \ + --manifest-path "$prototype_dir/Cargo.toml" --package "$package" +} + +build_package pyo3-comparison-base base +build_package pyo3-comparison-fs-capsule fs-capsule +build_package pyo3-comparison-fs-direct fs-direct +build_package pyo3-comparison-mime-capsule mime-capsule +build_package pyo3-comparison-mime-direct mime-direct + +mkdir -p "$stage" +cp "$target_root/base/release/libbase.so" "$stage/opendal_poc.so" +cp "$target_root/fs-capsule/release/libfs_capsule.so" "$stage/opendal_fs_capsule.so" +cp "$target_root/fs-direct/release/libfs_direct.so" "$stage/opendal_fs_direct.so" +cp "$target_root/mime-capsule/release/libmime_capsule.so" "$stage/opendal_mime_capsule.so" +cp "$target_root/mime-direct/release/libmime_direct.so" "$stage/opendal_mime_direct.so" + +"$audit" "$stage/opendal_poc.so" PyInit_opendal_poc +"$audit" "$stage/opendal_fs_capsule.so" PyInit_opendal_fs_capsule +"$audit" "$stage/opendal_fs_direct.so" PyInit_opendal_fs_direct +"$audit" "$stage/opendal_mime_capsule.so" PyInit_opendal_mime_capsule +"$audit" "$stage/opendal_mime_direct.so" PyInit_opendal_mime_direct + +if [ "$#" -eq 0 ]; then + set -- --all +fi + +PYTHONPATH="$stage" python3 "$prototype_dir/compare.py" "$@" From bf99950175a6af4b9b7ede2213d8303bcd4e03e9 Mon Sep 17 00:00:00 2001 From: Erick Guan <297343+erickguan@users.noreply.github.com> Date: Mon, 10 Aug 2026 10:03:15 +0800 Subject: [PATCH 8/8] feat(ruby): prototype shared runtime adapter --- .../dynamic-extensions/audit-elf-exports.sh | 9 +- bindings/ruby/dynamic-extensions/.gitignore | 16 + bindings/ruby/dynamic-extensions/Cargo.lock | 297 ++++++++++++++++ bindings/ruby/dynamic-extensions/Cargo.toml | 25 ++ bindings/ruby/dynamic-extensions/NOTES.md | 50 +++ bindings/ruby/dynamic-extensions/README.md | 40 +++ .../dynamic-extensions/adapter/Cargo.toml | 36 ++ .../ruby/dynamic-extensions/adapter/build.rs | 31 ++ .../dynamic-extensions/adapter/exports.map | 21 ++ .../dynamic-extensions/adapter/src/lib.rs | 329 ++++++++++++++++++ .../ruby/dynamic-extensions/ruby/example.rb | 28 ++ .../ruby/main/lib/opendal.rb | 37 ++ .../ruby/python_cross_check.py | 26 ++ .../service-fs/lib/opendal/services/fs.rb | 39 +++ .../ruby/dynamic-extensions/ruby/test_poc.rb | 82 +++++ .../ruby/dynamic-extensions/run-ruby-linux.sh | 70 ++++ 16 files changed, 1132 insertions(+), 4 deletions(-) create mode 100644 bindings/ruby/dynamic-extensions/.gitignore create mode 100644 bindings/ruby/dynamic-extensions/Cargo.lock create mode 100644 bindings/ruby/dynamic-extensions/Cargo.toml create mode 100644 bindings/ruby/dynamic-extensions/NOTES.md create mode 100644 bindings/ruby/dynamic-extensions/README.md create mode 100644 bindings/ruby/dynamic-extensions/adapter/Cargo.toml create mode 100644 bindings/ruby/dynamic-extensions/adapter/build.rs create mode 100644 bindings/ruby/dynamic-extensions/adapter/exports.map create mode 100644 bindings/ruby/dynamic-extensions/adapter/src/lib.rs create mode 100644 bindings/ruby/dynamic-extensions/ruby/example.rb create mode 100644 bindings/ruby/dynamic-extensions/ruby/main/lib/opendal.rb create mode 100644 bindings/ruby/dynamic-extensions/ruby/python_cross_check.py create mode 100644 bindings/ruby/dynamic-extensions/ruby/service-fs/lib/opendal/services/fs.rb create mode 100644 bindings/ruby/dynamic-extensions/ruby/test_poc.rb create mode 100755 bindings/ruby/dynamic-extensions/run-ruby-linux.sh diff --git a/bindings/python/dynamic-extensions/audit-elf-exports.sh b/bindings/python/dynamic-extensions/audit-elf-exports.sh index 59a84bbbd1ab..97845f6c514c 100755 --- a/bindings/python/dynamic-extensions/audit-elf-exports.sh +++ b/bindings/python/dynamic-extensions/audit-elf-exports.sh @@ -16,13 +16,14 @@ set -eu -if [ "$#" -ne 2 ]; then - echo "usage: $0 ARTIFACT EXPECTED_SYMBOL" >&2 +if [ "$#" -lt 2 ]; then + echo "usage: $0 ARTIFACT EXPECTED_SYMBOL..." >&2 exit 2 fi artifact=$1 -expected=$2 +shift +expected=$(printf '%s\n' "$@" | LC_ALL=C sort -u) actual=$( nm --dynamic --defined-only --extern-only --format=posix "$artifact" \ | awk '{ print $1 }' \ @@ -38,4 +39,4 @@ if [ "$actual" != "$expected" ]; then exit 1 fi -echo "$artifact exports only $expected" +echo "$artifact exports only the expected symbols" diff --git a/bindings/ruby/dynamic-extensions/.gitignore b/bindings/ruby/dynamic-extensions/.gitignore new file mode 100644 index 000000000000..d7d2adf88a85 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/.gitignore @@ -0,0 +1,16 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +/target/ diff --git a/bindings/ruby/dynamic-extensions/Cargo.lock b/bindings/ruby/dynamic-extensions/Cargo.lock new file mode 100644 index 000000000000..f6b68fb620cd --- /dev/null +++ b/bindings/ruby/dynamic-extensions/Cargo.lock @@ -0,0 +1,297 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "bindgen" +version = "0.72.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "993776b509cfb49c750f11b8f07a46fa23e0a1386ffc01fb1e7d343efc387895" +dependencies = [ + "bitflags", + "cexpr", + "clang-sys", + "itertools", + "proc-macro2", + "quote", + "regex", + "rustc-hash", + "shlex", + "syn", +] + +[[package]] +name = "bitflags" +version = "2.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b588b76d00fde79687d7646a9b5bdf3cc0f655e0bbd080335a95d7e96f3587da" + +[[package]] +name = "bytes" +version = "1.12.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" + +[[package]] +name = "cexpr" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6fac387a98bb7c37292057cffc56d62ecb629900026402633ae9160df93a8766" +dependencies = [ + "nom", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "clang-sys" +version = "1.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b023947811758c97c59bf9d1c188fd619ad4718dcaa767947df1cadb14f39f4" +dependencies = [ + "glob", + "libc", + "libloading", +] + +[[package]] +name = "either" +version = "1.17.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9e5e8f6c15a24b9a3ee5efec809ccd006d3b30e8b3bb63c39af737c7f87daa1d" + +[[package]] +name = "glob" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e4eba85ea1d0a966a983acd07deee566e67395d2d96b6fb39e62b5a833f1eb0b" + +[[package]] +name = "itertools" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186" +dependencies = [ + "either", +] + +[[package]] +name = "lazy_static" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe" + +[[package]] +name = "libc" +version = "0.2.189" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" + +[[package]] +name = "libloading" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7c4b02199fee7c5d21a5ae7d8cfa79a6ef5bb2fc834d6e9058e89c825efdc55" +dependencies = [ + "cfg-if", + "windows-link", +] + +[[package]] +name = "magnus" +version = "0.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3b36a5b126bbe97eb0d02d07acfeb327036c6319fd816139a49824a83b7f9012" +dependencies = [ + "bytes", + "magnus-macros", + "rb-sys", + "rb-sys-env", + "seq-macro", +] + +[[package]] +name = "magnus-macros" +version = "0.8.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47607461fd8e1513cb4f2076c197d8092d921a1ea75bd08af97398f593751892" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "memchr" +version = "2.8.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98" + +[[package]] +name = "minimal-lexical" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a" + +[[package]] +name = "nom" +version = "7.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a" +dependencies = [ + "memchr", + "minimal-lexical", +] + +[[package]] +name = "opendal-dynamic-extension-sdk" +version = "0.0.0" + +[[package]] +name = "opendal-ruby-runtime-poc" +version = "0.0.0" +dependencies = [ + "bytes", + "libloading", + "magnus", + "opendal-dynamic-extension-sdk", + "rb-sys", + "rb-sys-env", +] + +[[package]] +name = "proc-macro2" +version = "1.0.107" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "985e7ec9bb745e6ce6535b544d84d6cd6f7ad8bd711c398938ae983b91a766d9" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "quote" +version = "1.0.47" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fbf4db142a473a8d80c26bbf18454ed458bf8d26c8219c331daecfdbd079001" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "rb-sys" +version = "0.9.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "45ca28513560e56cfb79a62b1fce363c73af170a182024ce880c77ee9429920a" +dependencies = [ + "rb-sys-build", +] + +[[package]] +name = "rb-sys-build" +version = "0.9.128" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce04b2c55eff3a21aaa623fcc655d94373238e72cac6b3e1a3641ff31649f99a" +dependencies = [ + "bindgen", + "lazy_static", + "proc-macro2", + "quote", + "regex", + "shell-words", + "syn", +] + +[[package]] +name = "rb-sys-env" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cca7ad6a7e21e72151d56fe2495a259b5670e204c3adac41ee7ef676ea08117a" + +[[package]] +name = "regex" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f020237b6c8eed93db2e2cb53c00c60a8e1bc73da7d073199a1180401450218d" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.16" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fcfdb36bda0c880c5931cdc7a2bcdc8ba4556847b9d912bca70bc94708711ad" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "rustc-hash" +version = "2.1.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" + +[[package]] +name = "seq-macro" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "shell-words" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dc6fe69c597f9c37bfeeeeeb33da3530379845f10be461a66d16d03eca2ded77" + +[[package]] +name = "shlex" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64" + +[[package]] +name = "syn" +version = "2.0.119" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "872831b642d1a07999a962a351ed35b955ea2cfc8f3862091e2a240a84f17297" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" diff --git a/bindings/ruby/dynamic-extensions/Cargo.toml b/bindings/ruby/dynamic-extensions/Cargo.toml new file mode 100644 index 000000000000..14107bea1b15 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/Cargo.toml @@ -0,0 +1,25 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +members = ["adapter"] +resolver = "2" + +[workspace.package] +edition = "2024" +license = "Apache-2.0" +publish = false +rust-version = "1.91" +version = "0.0.0" diff --git a/bindings/ruby/dynamic-extensions/NOTES.md b/bindings/ruby/dynamic-extensions/NOTES.md new file mode 100644 index 000000000000..72979021d55c --- /dev/null +++ b/bindings/ruby/dynamic-extensions/NOTES.md @@ -0,0 +1,50 @@ + + +# Prototype Notes + +Question: Can Ruby use the same shared runtime and native extension artifacts +as Python without linking a binding-private OpenDAL or Tokio graph? + +Observed result: + +- The Ruby adapter negotiates runtime protocol 1 and rejects a binding that + requires protocol 2. +- Registering the FS Ruby package does not load its native library. Constructing + the first FS operator activates it. +- Ruby writes and reads an 8 KiB payload through the runtime-owned handle. The + adapter exercises the output-buffer resize contract and rejects use after + `Operator#close`. +- The Ruby adapter links Magnus, the protocol SDK, and the native loader. It + does not link `opendal-core`, a service crate, or Tokio. +- The same physical runtime and FS artifacts complete a second FS round trip + through the Python adapter. +- The runtime, FS extension, and Ruby adapter match their explicit ELF export + allowlists. Ruby requires both `Init_opendal_ruby_poc` and + `ruby_abi_version`. + +Production gate: the current runtime protocol owns service registration and an +operation callback table, but it has no layer registration, `LayerHandle`, or +operator-layer composition function. A successful FS run proves cross-language +runtime reuse only. It does not satisfy the design requirement that arbitrary +native layers preserve `apply_service` and `apply_context` semantics. + +Decision: do not migrate the production Python and Ruby bindings yet. The +current runtime implements the Design B operation-table boundary for services, +not the selected Design C shared OpenDAL graph. Production work first needs a +runtime-owned service/layer factory interface and a real operation that proves +an independently packaged native layer composes through that graph. diff --git a/bindings/ruby/dynamic-extensions/README.md b/bindings/ruby/dynamic-extensions/README.md new file mode 100644 index 000000000000..338a587c75fb --- /dev/null +++ b/bindings/ruby/dynamic-extensions/README.md @@ -0,0 +1,40 @@ + + +# Ruby Shared Runtime Prototype + +PROTOTYPE: delete or absorb this directory after the design question is +answered. + +Question: Can a Ruby binding adapter use the exact language-neutral runtime and +FS service artifacts used by the Python dynamic-extension POC? + +The Magnus adapter depends on the extension SDK and loader only. It does not +link OpenDAL core, an OpenDAL service, or Tokio. The runner builds and stages +the existing `opendal-runtime-poc` and `fs-extension` artifacts without +recompiling them for Ruby. + +Run the experiment on Linux: + +```console +./run-ruby-linux.sh +``` + +The runner verifies protocol negotiation, lazy FS activation, real FS +write/read, runtime-owned handle destruction, rejection of a newer binding +protocol, and final ELF export allowlists. It also stages the same runtime and +FS binaries under the Python adapter and completes a second real FS round trip. diff --git a/bindings/ruby/dynamic-extensions/adapter/Cargo.toml b/bindings/ruby/dynamic-extensions/adapter/Cargo.toml new file mode 100644 index 000000000000..618c18535b49 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/Cargo.toml @@ -0,0 +1,36 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[package] +name = "opendal-ruby-runtime-poc" +edition.workspace = true +license.workspace = true +publish.workspace = true +rust-version.workspace = true +version.workspace = true + +[lib] +crate-type = ["cdylib"] +name = "opendal_ruby_poc" + +[dependencies] +bytes = "1" +libloading = "0.8.9" +magnus = { version = "0.8", features = ["bytes"] } +opendal-dynamic-extension-sdk = { path = "../../../python/dynamic-extensions/sdk" } +rb-sys = { version = "0.9.110", default-features = false } + +[build-dependencies] +rb-sys-env = "0.2.3" diff --git a/bindings/ruby/dynamic-extensions/adapter/build.rs b/bindings/ruby/dynamic-extensions/adapter/build.rs new file mode 100644 index 000000000000..d41d49b50f90 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/build.rs @@ -0,0 +1,31 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::env; +use std::path::PathBuf; + +fn main() -> Result<(), Box> { + let _ = rb_sys_env::activate()?; + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("linux") { + let manifest_dir = PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap()); + let export_map = manifest_dir.join("exports.map"); + println!("cargo:rerun-if-changed={}", export_map.display()); + println!( + "cargo:rustc-cdylib-link-arg=-Wl,--version-script={}", + export_map.display() + ); + } + Ok(()) +} diff --git a/bindings/ruby/dynamic-extensions/adapter/exports.map b/bindings/ruby/dynamic-extensions/adapter/exports.map new file mode 100644 index 000000000000..f1e9b63961c7 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/exports.map @@ -0,0 +1,21 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +OPENDAL_RUBY_POC_1 { + global: + Init_opendal_ruby_poc; + local: + *; +}; diff --git a/bindings/ruby/dynamic-extensions/adapter/src/lib.rs b/bindings/ruby/dynamic-extensions/adapter/src/lib.rs new file mode 100644 index 000000000000..ac83522dff86 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/adapter/src/lib.rs @@ -0,0 +1,329 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to you under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashMap; +use std::ffi::c_void; +use std::mem::size_of; +use std::path::{Path, PathBuf}; +use std::ptr; +use std::sync::OnceLock; +use std::sync::atomic::{AtomicUsize, Ordering}; + +use libloading::{Library, Symbol}; +use magnus::prelude::*; +use magnus::{Error, RString, Ruby, function, method}; +use opendal_dynamic_extension_sdk::{ + ByteSlice, KeyValue, OutputBuffer, RuntimeApiV1, RuntimeProtocolInfoV1, + STATUS_BUFFER_TOO_SMALL, STATUS_OK, ServiceRegistrationV1, +}; + +type GetRuntimeApiFn = + unsafe extern "C" fn(u32, *mut RuntimeProtocolInfoV1, *mut *const RuntimeApiV1) -> i32; + +struct NativeRuntime { + _library: Library, + api_address: usize, + library_path: PathBuf, + minimum_protocol: u32, + protocol: u32, +} + +impl NativeRuntime { + fn api(&self) -> &RuntimeApiV1 { + unsafe { &*(self.api_address as *const RuntimeApiV1) } + } +} + +static RUNTIME: OnceLock = OnceLock::new(); + +fn runtime_error(ruby: &Ruby, message: impl AsRef) -> Error { + Error::new(ruby.exception_runtime_error(), message.as_ref().to_owned()) +} + +fn output(capacity: usize) -> (OutputBuffer, Vec) { + let mut storage = vec![0; capacity]; + let buffer = OutputBuffer { + data: storage.as_mut_ptr(), + capacity: storage.len(), + len: 0, + }; + (buffer, storage) +} + +fn output_message(buffer: &OutputBuffer, storage: &[u8]) -> String { + String::from_utf8_lossy(&storage[..buffer.len.min(storage.len())]).into_owned() +} + +fn byte_slice(value: &[u8]) -> ByteSlice { + ByteSlice { + data: value.as_ptr(), + len: value.len(), + } +} + +fn require_runtime(ruby: &Ruby) -> Result<&'static NativeRuntime, Error> { + RUNTIME + .get() + .ok_or_else(|| runtime_error(ruby, "OpenDAL runtime is not loaded")) +} + +fn load_runtime(ruby: &Ruby, library_path: String, required_protocol: u32) -> Result<(), Error> { + let library_path = Path::new(&library_path) + .canonicalize() + .map_err(|error| runtime_error(ruby, error.to_string()))?; + if let Some(runtime) = RUNTIME.get() { + if runtime.library_path != library_path { + return Err(runtime_error( + ruby, + format!( + "runtime already loaded from {}", + runtime.library_path.display() + ), + )); + } + if required_protocol < runtime.minimum_protocol || required_protocol > runtime.protocol { + return Err(runtime_error(ruby, "runtime protocol is incompatible")); + } + return Ok(()); + } + + let library = unsafe { Library::new(&library_path) } + .map_err(|error| runtime_error(ruby, error.to_string()))?; + let get_api: Symbol<'_, GetRuntimeApiFn> = unsafe { + library + .get(b"opendal_runtime_get_api_v1") + .map_err(|error| runtime_error(ruby, error.to_string()))? + }; + let mut protocol = RuntimeProtocolInfoV1 { + struct_size: size_of::(), + minimum_runtime_protocol: 0, + runtime_protocol: 0, + }; + let mut api = ptr::null(); + let status = unsafe { get_api(required_protocol, &mut protocol, &mut api) }; + if status != STATUS_OK || api.is_null() { + return Err(runtime_error( + ruby, + format!( + "runtime protocol negotiation failed: required={required_protocol}, supported={}..{}", + protocol.minimum_runtime_protocol, protocol.runtime_protocol + ), + )); + } + + RUNTIME + .set(NativeRuntime { + _library: library, + api_address: api as usize, + library_path, + minimum_protocol: protocol.minimum_runtime_protocol, + protocol: protocol.runtime_protocol, + }) + .map_err(|_| runtime_error(ruby, "runtime was loaded concurrently")) +} + +fn minimum_runtime_protocol(ruby: &Ruby) -> Result { + Ok(require_runtime(ruby)?.minimum_protocol) +} + +fn runtime_protocol(ruby: &Ruby) -> Result { + Ok(require_runtime(ruby)?.protocol) +} + +fn register_service( + ruby: &Ruby, + package_id: String, + component_id: String, + entry_symbol: String, + library_path: String, + required_protocol: u32, +) -> Result<(), Error> { + let runtime = require_runtime(ruby)?; + let package = byte_slice(package_id.as_bytes()); + let component = byte_slice(component_id.as_bytes()); + let entry = byte_slice(entry_symbol.as_bytes()); + let path = Path::new(&library_path) + .canonicalize() + .map_err(|error| runtime_error(ruby, error.to_string()))?; + let path = path.to_string_lossy(); + let path_slice = byte_slice(path.as_bytes()); + let registration = ServiceRegistrationV1 { + struct_size: size_of::(), + required_runtime_protocol: required_protocol, + package_id: package, + component_id: component, + entry_symbol: entry, + library_path: path_slice, + }; + let (mut error, error_storage) = output(4096); + let status = unsafe { (runtime.api().register_service)(®istration, &mut error) }; + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(()) +} + +#[magnus::wrap(class = "OpenDal::Operator", free_immediately)] +struct Operator { + handle_address: AtomicUsize, +} + +impl Drop for Operator { + fn drop(&mut self) { + self.close(); + } +} + +impl Operator { + fn close(&self) { + let handle_address = self.handle_address.swap(0, Ordering::AcqRel); + if handle_address == 0 { + return; + } + if let Some(runtime) = RUNTIME.get() { + unsafe { + (runtime.api().operator_destroy)(handle_address as *mut c_void); + } + } + } + + fn require_handle(&self, ruby: &Ruby) -> Result<*mut c_void, Error> { + let handle_address = self.handle_address.load(Ordering::Acquire); + if handle_address == 0 { + return Err(runtime_error(ruby, "operator is closed")); + } + Ok(handle_address as *mut c_void) + } + + fn new( + ruby: &Ruby, + scheme: String, + options: Option>, + ) -> Result { + let runtime = require_runtime(ruby)?; + let scheme = scheme.trim().to_lowercase().replace('_', "-"); + let options = options.unwrap_or_default(); + let pairs: Vec = options + .iter() + .map(|(key, value)| KeyValue { + key: byte_slice(key.as_bytes()), + value: byte_slice(value.as_bytes()), + }) + .collect(); + let mut handle = ptr::null_mut(); + let (mut error, error_storage) = output(4096); + let status = unsafe { + (runtime.api().create_operator)( + byte_slice(scheme.as_bytes()), + pairs.as_ptr(), + pairs.len(), + &mut handle, + &mut error, + ) + }; + if status != STATUS_OK || handle.is_null() { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(Self { + handle_address: AtomicUsize::new(handle as usize), + }) + } + + fn close_ruby(operator: &Self) { + operator.close(); + } + + fn info_json(ruby: &Ruby, operator: &Self) -> Result { + let runtime = require_runtime(ruby)?; + let (mut result, result_storage) = output(4096); + let (mut error, error_storage) = output(4096); + let status = unsafe { + (runtime.api().operator_info)(operator.require_handle(ruby)?, &mut result, &mut error) + }; + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(output_message(&result, &result_storage)) + } + + fn write(ruby: &Ruby, operator: &Self, path: String, data: RString) -> Result<(), Error> { + let runtime = require_runtime(ruby)?; + let (mut error, error_storage) = output(4096); + let data = data.to_bytes(); + let status = unsafe { + (runtime.api().operator_write)( + operator.require_handle(ruby)?, + byte_slice(path.as_bytes()), + byte_slice(&data), + &mut error, + ) + }; + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + Ok(()) + } + + fn read(ruby: &Ruby, operator: &Self, path: String) -> Result { + let runtime = require_runtime(ruby)?; + let (mut result, mut result_storage) = output(4096); + let (mut error, error_storage) = output(4096); + let mut status = unsafe { + (runtime.api().operator_read)( + operator.require_handle(ruby)?, + byte_slice(path.as_bytes()), + &mut result, + &mut error, + ) + }; + if status == STATUS_BUFFER_TOO_SMALL { + (result, result_storage) = output(result.len); + status = unsafe { + (runtime.api().operator_read)( + operator.require_handle(ruby)?, + byte_slice(path.as_bytes()), + &mut result, + &mut error, + ) + }; + } + if status != STATUS_OK { + return Err(runtime_error(ruby, output_message(&error, &error_storage))); + } + result_storage.truncate(result.len); + Ok(result_storage.into()) + } +} + +#[magnus::init(name = "opendal_ruby_poc")] +fn init(ruby: &Ruby) -> Result<(), Error> { + let opendal = ruby.define_module("OpenDal")?; + let runtime = opendal.define_module("Runtime")?; + runtime.define_singleton_method("load", function!(load_runtime, 2))?; + runtime.define_singleton_method( + "minimum_runtime_protocol", + function!(minimum_runtime_protocol, 0), + )?; + runtime.define_singleton_method("runtime_protocol", function!(runtime_protocol, 0))?; + runtime.define_singleton_method("register_service", function!(register_service, 5))?; + + let operator = opendal.define_class("Operator", ruby.class_object())?; + operator.define_singleton_method("new", function!(Operator::new, 2))?; + operator.define_method("close", method!(Operator::close_ruby, 0))?; + operator.define_method("info_json", method!(Operator::info_json, 0))?; + operator.define_method("write", method!(Operator::write, 2))?; + operator.define_method("read", method!(Operator::read, 1))?; + Ok(()) +} diff --git a/bindings/ruby/dynamic-extensions/ruby/example.rb b/bindings/ruby/dynamic-extensions/ruby/example.rb new file mode 100644 index 000000000000..5379ca2c5a81 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/example.rb @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "tmpdir" +require "opendal" +require "opendal/services/fs" + +Dir.mktmpdir("opendal-ruby-runtime-poc-") do |root| + operator = OpenDal::Operator.new("fs", {"root" => root}) + operator.write("hello.txt", "Hello from Ruby through the shared runtime!") + puts operator.info + puts operator.read("hello.txt") + operator.close +end diff --git a/bindings/ruby/dynamic-extensions/ruby/main/lib/opendal.rb b/bindings/ruby/dynamic-extensions/ruby/main/lib/opendal.rb new file mode 100644 index 000000000000..c0b9f775e1ad --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/main/lib/opendal.rb @@ -0,0 +1,37 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "json" +require_relative "opendal_ruby_poc" + +module OpenDal + REQUIRED_RUNTIME_PROTOCOL = Integer( + ENV.fetch("OPENDAL_POC_REQUIRED_RUNTIME_PROTOCOL", "1"), + 10 + ) + + Runtime.load( + File.expand_path("opendal/_native/libopendal_runtime_poc.so", __dir__), + REQUIRED_RUNTIME_PROTOCOL + ) + + class Operator + def info + JSON.parse(info_json) + end + end +end diff --git a/bindings/ruby/dynamic-extensions/ruby/python_cross_check.py b/bindings/ruby/dynamic-extensions/ruby/python_cross_check.py new file mode 100644 index 000000000000..24238c8aa48a --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/python_cross_check.py @@ -0,0 +1,26 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from tempfile import TemporaryDirectory + +import opendal.services.fs +from opendal import Operator + +with TemporaryDirectory(prefix="opendal-shared-artifact-poc-") as root: + with Operator("fs", root=root) as operator: + operator.write("python.txt", b"same runtime and FS artifacts") + assert operator.read("python.txt") == b"same runtime and FS artifacts" + +print({"adapter": "ctypes", "same_native_artifacts": True, "fs_round_trip": True}) diff --git a/bindings/ruby/dynamic-extensions/ruby/service-fs/lib/opendal/services/fs.rb b/bindings/ruby/dynamic-extensions/ruby/service-fs/lib/opendal/services/fs.rb new file mode 100644 index 000000000000..f8b6d641404e --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/service-fs/lib/opendal/services/fs.rb @@ -0,0 +1,39 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "opendal" + +module OpenDal + module Services + module Fs + MANIFEST = { + required_runtime_protocol: 1, + package_id: "opendal-service-fs-poc", + component_id: "fs", + native_entry_symbol: "opendal_service_fs_bootstrap_v1" + }.freeze + + Runtime.register_service( + MANIFEST.fetch(:package_id), + MANIFEST.fetch(:component_id), + MANIFEST.fetch(:native_entry_symbol), + File.expand_path("fs/_native/libfs_extension.so", __dir__), + MANIFEST.fetch(:required_runtime_protocol) + ) + end + end +end diff --git a/bindings/ruby/dynamic-extensions/ruby/test_poc.rb b/bindings/ruby/dynamic-extensions/ruby/test_poc.rb new file mode 100644 index 000000000000..b34844288765 --- /dev/null +++ b/bindings/ruby/dynamic-extensions/ruby/test_poc.rb @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# frozen_string_literal: true + +require "open3" +require "rbconfig" +require "tmpdir" +require "opendal" + +def assert(condition, message) + raise message unless condition +end + +def loaded?(basename) + File.read("/proc/self/maps").include?(basename) +end + +assert(OpenDal::Runtime.minimum_runtime_protocol == 1, "unexpected minimum protocol") +assert(OpenDal::Runtime.runtime_protocol == 1, "unexpected current protocol") +assert(!loaded?("libfs_extension.so"), "FS loaded before package registration") + +begin + OpenDal::Operator.new("fs", {}) + raise "unregistered FS construction succeeded" +rescue RuntimeError => error + assert(error.message.include?("not registered"), "unexpected registration error") +end + +require "opendal/services/fs" +assert(!loaded?("libfs_extension.so"), "FS loaded during package registration") + +Dir.mktmpdir("opendal-ruby-runtime-poc-") do |root| + operator = OpenDal::Operator.new("fs", {"root" => root}) + assert(loaded?("libfs_extension.so"), "FS did not load during construction") + payload = "ruby-runtime-poc-" * 512 + operator.write("hello.txt", payload) + assert(operator.read("hello.txt") == payload, "FS round trip failed") + assert(operator.info.fetch("scheme") == "fs", "unexpected operator scheme") + operator.close + + begin + operator.read("hello.txt") + raise "closed operator remained usable" + rescue RuntimeError => error + assert(error.message == "operator is closed", "unexpected closed-handle error") + end +end + +main = ENV.fetch("OPENDAL_RUBY_POC_MAIN") +env = {"OPENDAL_POC_REQUIRED_RUNTIME_PROTOCOL" => "2"} +_, stderr, status = Open3.capture3( + env, + RbConfig.ruby, + "-I#{main}", + "-ropendal", + "-e", + "abort 'incompatible runtime unexpectedly loaded'" +) +assert(!status.success?, "newer binding protocol unexpectedly loaded") +assert(stderr.include?("runtime protocol negotiation failed"), "missing protocol diagnostic") + +puts({ + adapter: "Magnus", + runtime_protocol: OpenDal::Runtime.runtime_protocol, + fs_lazy_load: true, + fs_round_trip: true, + closed_handle_rejected: true, + newer_protocol_rejected: true +}.inspect) diff --git a/bindings/ruby/dynamic-extensions/run-ruby-linux.sh b/bindings/ruby/dynamic-extensions/run-ruby-linux.sh new file mode 100755 index 000000000000..aa84027c7f6c --- /dev/null +++ b/bindings/ruby/dynamic-extensions/run-ruby-linux.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env sh +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +prototype_dir=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +shared_dir="$prototype_dir/../../python/dynamic-extensions" +target_root="$prototype_dir/target" +runtime_target="$target_root/shared-runtime" +fs_target="$target_root/shared-fs" +adapter_target="$target_root/ruby-adapter" +stage="$target_root/ruby-stage" + +if cargo tree --locked --offline --manifest-path "$prototype_dir/Cargo.toml" \ + --package opendal-ruby-runtime-poc \ + | rg -q 'opendal-core|opendal-service-|tokio'; then + echo "Ruby adapter unexpectedly links OpenDAL core, a service, or Tokio" >&2 + exit 1 +fi + +CARGO_TARGET_DIR="$runtime_target" cargo build --release --locked --offline \ + --manifest-path "$shared_dir/Cargo.toml" --package opendal-runtime-poc +CARGO_TARGET_DIR="$fs_target" cargo build --release --locked --offline \ + --manifest-path "$shared_dir/Cargo.toml" --package fs-extension +CARGO_TARGET_DIR="$adapter_target" cargo build --release --locked --offline \ + --manifest-path "$prototype_dir/Cargo.toml" --package opendal-ruby-runtime-poc + +runtime="$runtime_target/release/libopendal_runtime_poc.so" +fs="$fs_target/release/libfs_extension.so" +adapter="$adapter_target/release/libopendal_ruby_poc.so" + +"$shared_dir/audit-elf-exports.sh" "$runtime" opendal_runtime_get_api_v1 +"$shared_dir/audit-elf-exports.sh" "$fs" opendal_service_fs_bootstrap_v1 +"$shared_dir/audit-elf-exports.sh" \ + "$adapter" Init_opendal_ruby_poc ruby_abi_version + +mkdir -p \ + "$stage/main/lib/opendal/_native" \ + "$stage/fs/lib/opendal/services/fs/_native" \ + "$stage/python/main/opendal/_native" \ + "$stage/python/fs/opendal/services/fs/_native" +cp -R "$prototype_dir/ruby/main/." "$stage/main/" +cp -R "$prototype_dir/ruby/service-fs/." "$stage/fs/" +cp "$runtime" "$stage/main/lib/opendal/_native/" +cp "$adapter" "$stage/main/lib/opendal_ruby_poc.so" +cp "$fs" "$stage/fs/lib/opendal/services/fs/_native/" +cp -R "$shared_dir/python/main/." "$stage/python/main/" +cp -R "$shared_dir/python/service-fs/." "$stage/python/fs/" +cp "$runtime" "$stage/python/main/opendal/_native/" +cp "$fs" "$stage/python/fs/opendal/services/fs/_native/" + +RUBYLIB="$stage/main/lib:$stage/fs/lib" \ +OPENDAL_RUBY_POC_MAIN="$stage/main/lib" \ + ruby "$prototype_dir/ruby/test_poc.rb" +RUBYLIB="$stage/main/lib:$stage/fs/lib" ruby "$prototype_dir/ruby/example.rb" +PYTHONPATH="$stage/python/main:$stage/python/fs" \ + python3 "$prototype_dir/ruby/python_cross_check.py"