Chandra Shekhar Sirimala

This commit **prepares for the release of `google-cloud-storage` version 3.10.1**, primarily to deliver a **bug fix** for the `AsyncGrpcClient`. The fix addresses an issue where a `ValueError` was not raised when `api_endpoint` was unset while using `AnonymousCredentials`. This **maintenance release** updates version strings across `google-cloud-storage`'s core library, GAPIC client, and snippet metadata, ensuring users receive the improved client behavior.

This commit performs **maintenance** on the **`google-cloud-storage` testing infrastructure** to enable independent execution of bidirectional conformance tests. A new Nox session, `conftest_retry_bidi`, is introduced in `noxfile.py` for running these tests in isolation, alongside a refactoring of the existing `conftest_retry` session. Concurrently, the `test_bidi_reads.py` conformance test is updated to leverage `AsyncGrpcClient` and `AsyncAppendableObjectWriter`, which is essential for this new independent testing setup. This change improves the modularity and efficiency of running specific conformance tests within the library.

This commit introduces a **validation fix** within the **`AsyncGrpcClient`** of the `google-cloud-storage` library. It modifies the `__init__` method to **raise a `ValueError`** if `AnonymousCredentials` are used without explicitly providing an `api_endpoint`. This **prevents misconfiguration** and provides clearer error messages, ensuring that users attempting to connect to the Storage API anonymously must specify a target endpoint. New unit tests have been added to verify this essential validation behavior.

This commit initiates the **release process for `google-cloud-storage` version 3.10.0**, updating the library's internal versioning files, librarian state, and sample metadata. This **maintenance** release introduces a **new feature** for bucket encryption enforcement, significantly improves **performance** for rapid bucket reads and writes, and addresses several **bug fixes**. Notable fixes include preventing Windows blob download failures for specific names, resolving a **path traversal vulnerability** in `download_many_to_path`, and correcting token handling in rapid bucket operations. Merging this pull request will **trigger the official release** of the updated `google-cloud-storage` client library, making these enhancements and fixes available to users.

This commit introduces a **new feature** to the `google-cloud-storage` **samples**, specifically enhancing the `storage_transfer_manager_download_many.py` snippet. It integrates `argparse` support, enabling direct command-line execution and improving **CLI testing** capabilities for the `download_many_blobs_with_transfer_manager` function. The update also modifies the download logic to incorporate a blob name prefix and explicitly clarifies crucial aspects of **traversal safety** when handling multiple blob downloads. This significantly improves the sample's usability and provides clearer guidance for developers working with the Storage Transfer Manager.

This commit delivers a **bug fix** to the **`google-cloud-storage` client library** to enhance its robustness on Windows systems. It modifies the **`transfer_manager`** component to **skip downloading blobs whose names contain colon characters** (e.g., `C:`, `D:`) when the application runs on Windows, as these can lead to invalid path errors. A new `InvalidPathError` is introduced and raised by `_resolve_path` for such cases, which `download_many_to_path` now catches to issue a warning and continue processing other blobs. This prevents application crashes and ensures a more stable experience for **Windows users** performing bulk download operations.

This commit introduces an **enhancement** to the **`google.cloud.storage.transfer_manager`** module by modifying the return type of the `download_many_to_path` function. Previously, this function returned `None` for success or an `Exception` for failures; it now includes `UserWarning` objects in its return list to explicitly indicate when a file download was **skipped**. This new behavior provides clearer feedback to users when files are not downloaded due to reasons like `skip_if_exists=True` or invalid destination paths, improving the **observability and handling of bulk storage download operations**. The change is thoroughly validated with updated **unit and system tests** for `download_many_to_path`.

This commit implements a **temporary workaround** to disable mutual TLS (mTLS) specifically for the **Google Cloud Storage client library's Cloud Build zonal tests**. It modifies the `packages/google-cloud-storage/cloudbuild/run_zonal_tests.sh` script to set an environment variable that prevents mTLS usage. This **maintenance chore** addresses a known issue (`b/492452834`) that was causing test failures, ensuring the stability and successful execution of the zonal test suite until the underlying bug is resolved.

This commit delivers a **security bug fix** to the **`transfer_manager`** module within `google-cloud-storage`, specifically addressing a **path traversal vulnerability** in the `download_many_to_path` function. It introduces a new internal utility, `_resolve_path`, to strictly validate that all resolved download paths remain within the user-provided `destination_directory`. Blobs whose names would lead to paths outside this directory (e.g., via `../` or absolute paths) are now skipped with a warning, preventing directory traversal attacks. This is a **breaking change** for users who previously relied on writing files outside the intended target directory, as such downloads will no longer occur.

This commit **enables automated execution of bidirectional conformance tests** for the **Google Cloud Storage asynchronous client library** within the Kokoro CI system. It introduces the **new capability** to create **anonymous gRPC connections** in `async_grpc_client.py`, which is essential for testing, and includes **bug fixes** to prevent infinite retry loops by correctly handling stream closures on broken connections in `AsyncAppendableObjectWriter` and `AsyncMultiRangeDownloader`. The changes also involve **refactoring** the conformance test suite to be discoverable by `pytest` and adding necessary test utilities, significantly improving the reliability and coverage of **bidirectional streaming functionality**.

This commit **introduces a new system test** to validate the **Google Cloud Storage client library's** functionality with **cross-region GCS buckets**. The `test_basic_wrd_x_region` test specifically verifies **write, read, and delete operations** for buckets spanning different regions, enhancing the **robustness** of the client library. This **new feature (test)** also includes a minor **refactoring** in `async_multi_range_downloader.py` to add an internal retry counter and update the retry policy's error callback. The work ensures the client library's compatibility and reliability in complex multi-region storage environments by expanding test coverage.

This commit **fixes critical bugs** within the **`google-cloud-storage`** library's **asynchronous I/O streams**. It primarily addresses an issue where `x-goog-request-params` were incorrectly delimited with a comma instead of an ampersand in `async_read_object_stream.py` and `async_write_object_stream.py`, leading to malformed API requests. Additionally, a separate **bug fix** in `async_appendable_object_writer.py` ensures `current_metadata` is correctly passed during append operations. This work guarantees proper request formatting and metadata handling, improving the reliability of asynchronous object interactions with Google Cloud Storage.

This commit **fixes** issues in the **Google Cloud Storage microbenchmarks** by optimizing resource initialization and CPU core management. It **refactors** the multi-process write tests in `packages/google-cloud-storage/tests/perf/microbenchmarks/writes/test_writes.py` to initialize the multiprocessing pool and gRPC client only once per benchmark run, preventing skewed throughput calculations and process hangs. Additionally, it introduces `os.sched_setaffinity` to pin worker processes to specific CPU cores, which helps avoid hard IRQ interference from the NIC. This **performance improvement** for the testing infrastructure ensures more accurate and stable benchmark results.

This commit introduces **new time-based microbenchmarks** for **Google Cloud Storage read operations**, significantly expanding the project's **performance testing suite**. It adds comprehensive tests for multi-process and multi-coroutine scenarios, alongside new configuration and parameter definitions for these benchmarks. Furthermore, the **benchmark reporting utilities** in `_utils.py` are enhanced with functions for network interface and IRQ affinity, and updated to include new throughput calculations, providing deeper insights into client library performance. This **new capability** enables more granular and realistic analysis of read performance under various concurrent loads.

This commit delivers a crucial **performance fix** for the **asynchronous appendable object writing** functionality within the `google-cloud-storage` library. It **refactors** the `async_appendable_object_writer` to prevent excessive flushing on every append, which previously caused significant performance degradation. The `writes_resumption_strategy` is updated to correctly trigger state lookups only when a flush is explicitly requested, and a necessary check for `persisted_size` is added in `async_write_object_stream`. This optimization significantly **improves the efficiency** of append operations by reducing unnecessary network calls and state management overhead, with updated samples and comprehensive unit tests validating the new logic.

This commit performs a **maintenance update** by **removing support for Python 3.9** within the `google-cloud-storage` package. It updates the project's CI/CD configurations and local development environment to reflect this change. Specifically, GitHub branch protection rules are adjusted, Kokoro presubmit configurations are updated to target Python 3.10, and Nox test configurations for unit and system tests are modified to no longer include Python 3.9. This ensures that all future development, testing, and continuous integration for `google-cloud-storage` will exclusively target Python 3.10 and newer versions.

This commit performs a **maintenance update** for the **`google-cloud-storage`** library, bumping its version from `3.8.0` to `3.9.0`. This **minor version release** introduces several **new features** and **bug fixes**, significantly enhancing the library's capabilities. Notable additions include expanded functionality for the **async gRPC client** (e.g., `get_object`, `DELETE_OBJECT`), improved support for **appendable objects** and write handles, and the graduation of **Zonal Buckets** features from experimental status. It also includes various bug fixes related to async streams and write handle updates. This update prepares the library for a new release, offering users improved performance, stability, and a broader set of storage interaction options.

This commit **enhances documentation** for the **Google Cloud Storage samples**, specifically focusing on the **zonal buckets** feature. It introduces a new `README.md` file at `packages/google-cloud-storage/samples/snippets/zonal_buckets/` to provide detailed prerequisites and usage instructions for these specific snippets. Concurrently, the main `packages/google-cloud-storage/samples/README.md` is updated to include references and guidance for running the zonal buckets samples. This **documentation update** improves the discoverability and ease of use for developers exploring Google Cloud Storage's zonal buckets capabilities.

This commit introduces a **new capability** by exposing the `DELETE_OBJECT` operation within the **`AsyncGrpcClient`** for the `google-cloud-storage` package. It adds an asynchronous `delete_object` method to `async_grpc_client.py`, allowing users to programmatically delete objects using the gRPC client. This new functionality is thoroughly validated with both a new unit test in `test_async_grpc_client.py` and a system test in `test_zonal.py`. Consequently, users of the `AsyncGrpcClient` can now perform **asynchronous object deletion**, enhancing the client's capabilities for managing Google Cloud Storage resources.

This commit **promotes** the **Zonal Buckets** features within the `google-cloud-storage` library from their experimental `_experimental` namespace to stable, public locations. It involves **refactoring** the module structure by moving core **async I/O** components like `AsyncAppendableObjectWriter`, `AsyncGrpcClient`, `AsyncMultiRangeDownloader`, and various stream and retry managers. The old experimental paths are now **deprecated**, and users will need to **update their import statements** to reflect the new stable module locations. This change signifies the **stabilization** of these advanced storage capabilities, ensuring broader adoption and support for **asynchronous operations** with Zonal Buckets.