Gapic deduplication experimentation

[hebaalazzeh]

GAPIC Client Deduplication (Shared Utility Layer)

Goal

Reduce package size and improve maintainability of generated GAPIC clients by deduplicating redundant static helper methods and configuration logic into a shared layer.

Background

Currently, every generated service client (e.g., in google-cloud-compute, google-cloud-aiplatform) contains hundreds of lines of identical static methods for:

• mTLS endpoint resolution (_get_default_mtls_endpoint)
• Client certificate usage checks (_use_client_cert_effective)
• Environment variable reading (_read_environment_variables)

This duplication inflates package size and makes horizontal fixes difficult.

Proposed Approach (Shared Utility Layer)

We are introducing a shared_utils.py module within the generated services package. Common logic is extracted into this module. The generated service clients are refactored to delegate to these shared functions via thin pass-through stubs.

Benefits:

• Significant LOC Reduction: Thousands of lines of code saved across massive libraries.
• Backward Compatibility Preserved: Backward-compatible stubs on the client class ensure that advanced users monkey-patching "private" methods continue to function.
• Type Safety: Maintains strict type hints.

Manual Validation Results (google-cloud-kms)

A manual experiment was conducted on google-cloud-kms to validate this approach before updating the generator templates.

Steps Taken:

1. Created google/cloud/kms_v1/services/shared_utils.py.
2. Moved get_default_mtls_endpoint, use_client_cert_effective, and read_environment_variables to shared_utils.
3. Refactored KeyManagementServiceClient and EkmServiceClient to delegate to shared_utils.

Impact:

• Replaced ~35 lines of duplicate logic per client with one-line delegations.
• No functional regressions.

Testing Note (Environmental Edge Case Discovered):

During testing on Cloudtop environments, tests assuming default False states for client certs might fail if CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE=true is set globally. This is an artifact of the testing environment assumptions, not the deduplication logic. Overriding this variable during testing (CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE=false) confirms logic is sound.

References

• Design Doc

[gemini-code-assist]

Code Review

This pull request introduces shared utilities for mTLS endpoint handling in the google-cloud-kms package and adds an import profiling toolset under scripts/import_profiler/, which includes a profiler script, documentation, and a script to rewrite __init__.py files for lazy loading. The review feedback suggests several robust improvements to the profiling scripts: converting the rglob generator to a list to prevent traversal issues during directory deletion, updating the __all__ regex parser to support lists in addition to tuples, adding boundary checks to prevent potential IndexError during multi-line import parsing, and implementing __dir__ alongside __getattr__ to support autocompletion for lazy-loaded modules.

[gemini-code-assist]

Modifying the directory structure (deleting __pycache__ directories) while traversing it with rglob can lead to unexpected behavior or FileNotFoundError. Converting the generator to a list first ensures safe traversal.

Suggested change

	for p in pathlib.Path('.').rglob('__pycache__'):
	for p in list(pathlib.Path('.').rglob('__pycache__')):

[gemini-code-assist]

The __all__ variable in Python can be defined as either a tuple or a list. Updating the regular expression to support both parentheses and square brackets prevents parsing failures.

Suggested change

	all_match = re.search(r'__all__\s*=\s*\((.*?)\)', content, re.DOTALL)
	all_match = re.search(r'__all__\s*=\s*[(\[](.*?)[)\]]', content, re.DOTALL)

[gemini-code-assist]

If a multi-line import is malformed or lacks a closing parenthesis, iterating over lines without checking bounds can raise an IndexError. Adding a boundary check ensures safe parsing.

Suggested change

	if "(" in names_part and ")" not in names_part:
	# multi-line import
	import_names_str = names_part.replace("(", "")
	i += 1
	while ")" not in lines[i]:
	import_names_str += lines[i]
	i += 1
	import_names_str += lines[i].replace(")", "")
	if "(" in names_part and ")" not in names_part:
	# multi-line import
	import_names_str = names_part.replace("(", "")
	i += 1
	while i < len(lines) and ")" not in lines[i]:
	import_names_str += lines[i]
	i += 1
	if i < len(lines):
	import_names_str += lines[i].replace(")", "")

[gemini-code-assist]

When implementing lazy loading via __getattr__ (PEP 562), it is highly recommended to also define __dir__ so that interactive environments (like IPython, Jupyter, or standard REPL) and IDEs can correctly autocomplete the lazy-imported attributes.

    def __getattr__(name):
        if name in _LAZY_IMPORTS:
            module = importlib.import_module(_LAZY_IMPORTS[name], package=__name__)
            attr = getattr(module, name)
            globals()[name] = attr
            return attr
        raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

    def __dir__():
        return sorted(list(globals().keys()) + list(_LAZY_IMPORTS.keys()))