refactor(storage): Migrate conformance and system tests to StorageTransport with enhanced retry logic

[thiyaguk09]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the internal conformance tests by migrating them to use the StorageTransport class. This change standardizes how HTTP requests are made and how retries are handled, moving away from direct client methods to a more unified transport layer. The update also introduces more robust retry logic, ensuring that requests are only retried when safe and appropriate, based on HTTP methods and preconditions. This improves the reliability and maintainability of the test suite and the underlying client library.

Highlights

• Conformance Test Migration: Refactored conformance tests to directly utilize the StorageTransport class, centralizing HTTP request handling and retry logic for consistency.
• Enhanced Retry Logic: Updated the default retryable error function to intelligently consider HTTP methods and the presence of preconditions (e.g., ifGenerationMatch) when determining if a request should be retried.
• CI/CD Streamlining: Removed the 'docs' job and its corresponding status check from the CI/CD configuration, simplifying the build process.
• Test Bench Update: Upgraded the Storage Testbench Docker image version to v0.60.0 for improved testing capabilities.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request refactors Google Cloud Storage conformance tests to directly use StorageTransport.makeRequest for API calls, bypassing higher-level client methods, and updates the testbench Docker image. It also removes the docs CI job and related settings. Review comments highlight a bug in the create function's bucketExists check, where a 404 error was not handled, causing premature exit, and suggest an optimization to reuse the Gaxios instance in the authClient.request mock for efficiency.

[gemini-code-assist]

Creating a new Gaxios instance for every authClient.request call within this mock can be inefficient, especially if this mock is called frequently. While this is a test utility, reusing a single Gaxios instance might offer a slight performance improvement and better reflect how Gaxios is typically used in production code.

Consider initializing gaxios once outside the request function and reusing it.

authClient.request = async opts => {
  const gaxios = new Gaxios();
  return gaxios.request(opts);
};

[thiyaguk09]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the storage library to improve conformance testing, ensure Node 18 compatibility, and enhance retry logic with granular idempotency checks. Key updates include transitioning to full REST API paths and refactoring conformance tests to use raw API calls. Reviewers identified a bug in a setTimeout callback and a potential crash in File.isPublic due to unnormalized callbacks. Additionally, feedback suggests refining the decompress option handling in StorageTransport, avoiding response data pollution when attaching metadata, and optimizing performance by reducing unnecessary JSON parsing during retry evaluations.

[gemini-code-assist]

Hardcoding decompress: false here overrides any decompress option passed in reqOpts. While this might be intended to avoid Gaxios's automatic decompression in favor of manual handling, it prevents users from explicitly requesting decompression for non-stream requests or when they want Gaxios to handle it. Consider only setting it to false if not explicitly provided in reqOpts.

Suggested change

	// eslint-disable-next-line @typescript-eslint/no-explicit-any
	...({decompress: false} as any),
	// eslint-disable-next-line @typescript-eslint/no-explicit-any
	decompress: (reqOpts as any).decompress ?? false,

[gemini-code-assist]

Attaching headers and status directly to the data object can have unexpected side effects, especially if data is an Array, a Buffer, or a frozen object. It also pollutes the response data with metadata that might conflict with actual data fields. A cleaner approach would be to return a structured object or ensure data is a plain object before modification.

[gemini-code-assist]

Attempting to JSON.parse(data) on every retryable error where data is a string can be a significant performance bottleneck if the request body is a large non-JSON string (e.g., a large file upload). Consider checking if the request was intended to be JSON or limiting the size of the string being parsed.