Add S3 PSDrive provider and new Mount-S3PSDrive/Dismount-S3PSDrive cmdlets to AWS.Tools.S3 - #423
Add S3 PSDrive provider and new Mount-S3PSDrive/Dismount-S3PSDrive cmdlets to AWS.Tools.S3#423andliao445 wants to merge 9 commits into
Conversation
77ce443 to
4a1344b
Compare
|
In Suggestions: Map your existing regions onto files like this. (You can explore other ways or can decide on the split structure) This improves code readability and also helps a reviewer who wants to understand |
The current changes look good, but can you update the structure to somewhat like this? A subfolder for all the |
5b0ef2b to
52267e8
Compare
3735a02 to
54e61d0
Compare
| { | ||
| // Empty objects can reject the SDK's initial ranged GET. Fall back to the | ||
| // ordinary stream path; there is nothing to parallelize for an empty object. | ||
| return tu.OpenStreamAsync(new TransferUtilityOpenStreamRequest |
There was a problem hiding this comment.
this seems like a bug in the transfer utility. i dont think we should do this fallback behavior here
There was a problem hiding this comment.
I agree this probably belongs in TransferUtility, but without this fallback Get-Content breaks for zero-byte objects because the SDK’s ranged discovery can return InvalidRange before it falls back to a normal stream.
There was a problem hiding this comment.
let me see if i can fix this in transfer utility
There was a problem hiding this comment.
| // Default part size for Set-Content's non-seekable stream uploads. TU cannot choose from | ||
| // the final object length, so this keeps streams under S3's 10,000-part limit until ~156 GiB. | ||
| // Set-Content -PartSize overrides it. | ||
| private const long DefaultMultipartUploadPartSize = 16L * 1024 * 1024; |
There was a problem hiding this comment.
what is this here? doesnt seem like its used?
There was a problem hiding this comment.
It's used at S3Provider.cs:200 but the github diff just cut it off I think. Also at S3Provider.Content.cs:298
2a64e28 to
f8ddf69
Compare
| // Set-Content -PartSize overrides it. | ||
| private const long DefaultMultipartUploadPartSize = 16L * 1024 * 1024; | ||
|
|
||
| private Amazon.S3.Transfer.TransferUtility TransferUtilityForBucket(S3DriveInfo drive, string bucket) => |
There was a problem hiding this comment.
where is TransferUtilityForBucket used though
There was a problem hiding this comment.
in GetContentReader at S3Provider.Content.cs:97 and GetContentWriter at S3Provider.Content.cs:322
b19e87c to
9fe9118
Compare
|
During testing, found a potential data loss issue. Consider rejecting |
16a649e to
0f2ea93
Compare
| while (!string.IsNullOrEmpty(token)); | ||
|
|
||
| // Also delete the object at the exact key: when a name is both folder ("key/...") and | ||
| // object ("key"), the sweep above only covers "key/", leaving the shadowed object behind. |
There was a problem hiding this comment.
When recursively removing the folder data/, this also deletes an object literally named data if one exists as separate, unrelated content and since ShouldProcess only prompted for the prefix, a -Confirm'd user wouldn't see that extra object being deleted.
Fine as a deliberate delete, but wanted to flag it since it can remove data the user didn't ask for. Your call on whether to gate it or throw a warning about the object also being deleted with the same key name.
0f2ea93 to
6ef5825
Compare
Introduces the AWS.S3 PowerShell provider so buckets, prefixes, and objects can be browsed with the standard navigation commands, plus the Mount/Dismount-S3PSDrive cmdlets. Extends the advanced cmdlet scanner to recognize member-access verb/noun pairs, and adds the integration test suite for the provider.
…w pass Splits the cmdlets, parameters, and provider into one type per file and moves the provider partials under a Provider folder. Adds a configurable upload part size and support for pipeline-bound Set-Content writer parameters. Folds in the first round of review feedback (capability checks, region fallback, pagination, drive safety) and makes the tests run headless in CI, tagging the expensive live tests so they stay out of the daily pipeline.
…ect handling Resolves provider-qualified paths across multiple mounted drives and switches reads to the TransferUtility multipart stream. Surfaces the guided SSO login error and re-resolves rotated profile credentials. Makes folders win over colliding objects, returns a single item for Get-Item on the drive root, adds -Filter as a leaf-name wildcard, gives a consistent error for unsupported operations, validates -StorageClass at mount, and uses the OS-native separator in path errors.
…tion Adds the download part size parameter with validation and the Get-Help content for the cmdlets. Makes dismount errors deterministic and fixes profile rotation resolution.
… cleanup Fixes the TransferUtility part size handling and removes the zero-byte object fallback now that the SDK handles it. Updates the part-size test for the 5 MiB default and shortens the code comments to match the team's style.
Set-Content used to feed TransferUtility a non-seekable bridge stream for every write, which forced TU's multipart path (initiate + part + complete) even for a few KB. The writer now buffers content and decides at Close: under 5 MiB it hands TU a seekable MemoryStream so TU does a single PutObject, and at or over 5 MiB it escalates to the streaming multipart bridge, which is the old behavior unchanged. 5 MiB is S3's smallest multipart part and is under TU's 16 MiB MinSizeBeforePartUpload, so a seekable stream below it is guaranteed to become one PutObject. The threshold is a fixed internal bound, not the user's -PartSize, so the buffering path stays memory-bounded. The simple upload sets AutoResetStreamPosition so the SDK rereads the buffer from the start on the first send and any retry. Adds a "Small-write single PutObject" test context that checks the upload path by the stored object's ETag (dashless = one PutObject, dashed = multipart) across 0 B to 10 MiB, plus -StorageClass forwarding and escalation at exactly 5 MiB. Also updates the now-stale GetContentWriter comment that said every upload goes multipart. Small-write median drops from ~693 ms to ~314 ms, closing the gap to Write-S3Object from 2.6x to 1.2x.
Remove-Item on the S3 drive advertises -Filter (ProviderCapabilities.Filter) and the listing path honors it, but the delete path ignored it: a recursive delete listed every key under the prefix and deleted them all. So Remove-Item <prefix> -Filter *.log -Recurse deleted every object, including the ones the filter was meant to spare. That's a data-loss bug. Apply the existing MatchesFilter(LeafName(...)) in the recursive delete loop, the shadowed-exact-key delete, and the single-object path, matching how Get-ChildItem -Filter -Recurse scopes by leaf name. With no filter set, MatchesFilter returns true, so unfiltered deletes still remove everything. Adds two tests: a filtered recursive delete leaves non-matching and nested objects intact, and an unfiltered recursive delete still removes everything.
A recursive Remove-Item on a prefix already deletes an object sharing the prefix's name (folder-wins hides it from reads and single-level listings, so the exact-key sweep would otherwise orphan it). It now HEAD-checks that key and, when a real same-named object exists, warns that both were removed, so a -Confirm user who approved removing the folder sees the extra deletion.
…und" ItemExists, IsItemContainer, and HasChildItems caught NotFound and AccessDenied but let invalid/expired-credential errors propagate. The provider engine turns any throw from those into "Cannot find path ... does not exist", so an expired token surfaced as a bogus not-found. Catch IsInvalidCredentials in all three (like AccessDenied) so path resolution succeeds and the actual operation surfaces the genuine error. Mount still fails fast: ValidateRoot lets these propagate.
6ef5825 to
e3d7ee3
Compare
4 participants
Description
Adds an S3 PowerShell drive provider (
AWS.S3) to AWS.Tools.S3. AfterMount-S3PSDrive -Name S3, users navigate and operate on S3 with the standardprovider cmdlets —
Set-Location,Get-ChildItem,Get-Item,Get-Content,Set-Content,Remove-Item, andDismount-S3PSDrive— across buckets, prefixes(folders), and objects (files).
Highlights:
NavigationCmdletProvider+IContentCmdletProviderthat calls the AWS SDKdirectly (it does not invoke the S3 cmdlets).
existence-probe cache that collapses the engine's repeated per-command
path-resolution probes.
TransferUtility(multipart upload/download); Ctrl+C cancels anin-flight transfer.
optional
-Rootscopes a drive to a bucket or bucket+prefix.(
$StoredAWSCredentials/$StoredAWSRegion), with explicit drive parameterstaking precedence; credentials then fall back to the SDK default chain.
-StorageClassdrive default plus per-upload override;-Encoding,-AsByteStream, and-NoNewlineon content operations.Also included:
via a
VerbsData.*member-access expression (needed forMount-S3PSDrive;the scanner previously handled only string-literal verbs/nouns).
S3ItemInfotable format view forGet-ChildItemoutput.Motivation and Context
Lets users browse and manipulate S3 as a mounted drive — matching the built-in
FileSystem provider experience — instead of composing individual
Get-S3Object/Write-S3Objectcalls.Testing
Live Pester integration tests in
tests/S3/S3.PSDrive.Tests.ps1, run against realS3 (us-east-1, plus us-west-2 for the cross-region case): root/bucket/prefix
listing with pagination,
Get-Itemon all three item kinds, content round-trips(text and byte, encodings, BOM,
-Raw,-NoNewline), single and recursiveRemove-Item, storage-class default + per-upload override,-Rootprefixmounting, session-default credential/region fallback, cache-invalidation and
existence-probe correctness, name-collision handling, and unsupported-op
rejection (Copy-Item, Add-Content). The suite rides the shared repo harness
(
test-runnerprofile,Smoketag) like the siblingtests/S3/*.Tests.ps1.Dry-run
Breaking Changes Assessment
No breaking changes. This is purely additive — two new cmdlets and a new provider
in AWS.Tools.S3; no existing cmdlet, parameter, or behavior is modified. The
generator scanner change is backward-compatible: it adds handling for a new
attribute-argument shape (member access), leaving existing string-literal cmdlets
unaffected.
Screenshots (if appropriate)
N/A
Types of changes
Checklist
New/existing dependencies impact assessment, if applicable
No new dependencies. The provider uses the AWSSDK.S3 / AWSSDK.Core assemblies
AWS.Tools.S3 already references.
License