Priti Chattopadhyay

This commit introduces the **Parallel Composite Upload (PCU)** capability to the **storage writer**, enabling more efficient handling of large object uploads. It implements the core PCU logic in `storage/pcu.go` for concurrent part management and integrates it into the `storage/writer.go` to conditionally leverage this new functionality. A **bug fix** in `storage/grpc_client.go` ensures proper CRC32C checksum handling during object composition, which is critical for the integrity of PCU operations. Extensive **new tests** across `integration_test.go`, `pcu_test.go`, and `writer_test.go` validate the PCU's behavior, concurrency, and error handling.

This commit provides a **bug fix** for a **deadlock** issue observed in the **`storage`** module's `gRPCOneshotBidiWriteBufferSender` when handling high data volumes. It **decouples the send and receive loops** within the `connect` method of `gRPCOneshotBidiWriteBufferSender` and related gRPC writers to ensure independent operation, preventing resource contention. Additionally, a new helper function, `pickStreamError`, is introduced to prioritize stream errors, enhancing the robustness of gRPC write operations. This change improves the **reliability** and **performance** of gRPC-based storage interactions, with new **test cases** added to validate the improved stream error handling and prevent future deadlocks.

This commit performs **maintenance** by preparing for the **release of the `storage` client library version `v1.61.2`**. It updates the internal version constant in `storage/internal/version.go` and modifies the `librarian` state configuration in `.librarian/state.yaml` to reflect the new version. Additionally, the client library version is updated within the generated snippet metadata for `storage` control plane APIs. Merging this pull request will **auto-trigger the official release** of `google-cloud-go/storage` `v1.61.2`, making the updated library available to users.

This commit **reverts** a prior "librarian release" chore, effectively undoing updates to version numbers across the project. It specifically **rolls back** the internal version constant for the **storage library** in `storage/internal/version.go`, reverts the client library version in `snippet_metadata.google.storage.control.v2.json` for generated snippets, and restores the storage library version in `.librarian/state.yaml`. This **maintenance revert** ensures that the project's dependencies and reported versions for the **storage module** remain at their state before the attempted release, preventing potential instability or incorrect version reporting.

chore(storage): fix dependencies (#14156)

This commit performs a **maintenance update** for the **Google Cloud Storage client library**, incrementing its version from `v1.61.0` to `v1.61.1`. As part of an automated release preparation, it updates the internal version constant in `storage/internal/version.go`, the Librarian state in `.librarian/state.yaml`, and the client library version within snippet metadata. This **version bump** ensures consistency across internal tooling and prepares for the official release of the updated `storage` module.

This commit introduces a **new diagnostic capability** within the **storage gRPC client** to report on DirectPath utilization. It adds a new module, `storage/grpc_dp_diag.go`, which provides functions like `directPathDiagnostic` to identify and report reasons why gRPC DirectPath might not be used. This diagnostic information is then incorporated into gRPC request headers by modifying the client's metadata preparation in `storage/grpc_client.go`. This enhancement significantly improves **observability and debugging** for DirectPath connections, enabling better understanding and troubleshooting of network path selection issues.

This commit **refactors** the **Parallel Composite Upload (PCU)** configuration within the **storage** module, driven by a new design decision. The changes simplify configuration options, rename `numWorkers` to `maxConcurrency` for improved clarity, and extract buffer pool settings into a new `pcuSettings` struct. This **maintenance** effort enhances the maintainability and understanding of the PCU setup, with `pcu_test.go` updated to validate the revised configuration and remove deprecated test cases.

This commit primarily **fixes a potential nil pointer dereference** within the **`storage` module's `grpc_writer` component**. A nil check for `w.attrs` is added in the `Write` method of `storage/grpc_writer.go` to prevent crashes when accessing its MD5 field for checksum calculation. Concurrently, the **`TestGRPCWriter_MemoryAllocationPaths` test** in `storage/grpc_writer_test.go` is **improved and updated** with new parameters, a renamed mock sender, and refined zero-copy assertion logic. This work ensures the stability of the `grpc_writer` and enhances the reliability of its memory allocation tests.

This commit introduces significant **performance optimizations** to the **`storage` subsystem's gRPC writer** by implementing **zero-copy data transfer** and **lazy buffer allocation**. These enhancements, primarily within `grpc_writer.go` affecting operations like `OpenWriter` and `zeroCopyWrite`, are designed to **reduce memory consumption** and **improve data transfer efficiency** for gRPC operations. New tests in `grpc_writer_test.go` have been added to thoroughly verify the correctness of these memory allocation and zero-copy paths.

This commit introduces a **new capability** for the **storage subsystem** by implementing **Parallel Composite Uploads (PCU)**, designed for efficient handling of very large objects. It enables **asynchronous chunking and uploading** of data parts via worker goroutines, and performs **recursive multi-level composition** using `pcuState.composeParts()` to overcome object component limits. The implementation also includes **robust cleanup** logic, orchestrated by `pcuState.doCleanup()`, to reliably remove all temporary parts and intermediate objects upon successful completion or error, significantly enhancing the reliability and resource management for large object uploads.

This commit introduces a **new conformance test** within the **storage** module. The test case is designed to specifically verify the correct handling of **ranged reads with mid-stream retries for large objects**. This **testing** effort enhances the quality assurance for the storage system, ensuring its robustness and reliability when dealing with complex read operations and transient network issues.

This commit **fixes a test** within the **`storage` module** by adjusting the test cases for **GRPC routing interceptors**. Specifically, it modifies the expected header values and refines the metadata assertion logic in `storage/grpc_client_test.go`. This **test maintenance** ensures that the routing interceptor functionality of the GRPC client is correctly validated. The change improves the accuracy and reliability of the test suite, preventing false negatives or positives for future modifications to the **`storage` GRPC client**.

This commit performs a **maintenance chore** by **removing the implicit fallback allowed option for direct connectivity** within the **storage gRPC client**. It eliminates the `forceDirectConnectivityFallbackAllowed` constant and refactors the `prepareDirectPathMetadata` function. Now, the direct connectivity header is only injected if a value is explicitly set, preventing an automatic fallback option from being included by default. This change **streamlines direct connectivity configuration** by ensuring more explicit control over connection behavior and removing an unnecessary default.

This commit introduces a **new capability** for the **storage client** by adding an **experimental client option**, `WithDirectConnectivityEnforced`. This option allows users to explicitly request **direct path gRPC connectivity** for storage operations. The implementation involves new gRPC routing interceptors within `storage/grpc_client.go` that inject a `force_direct_connectivity` gRPC header based on the client's configuration. This enhancement provides finer control over network routing for storage requests, potentially optimizing performance or bypassing intermediate proxies.

This commit provides a **bug fix** for a **deadlock** occurring in the **`storage` package's multi-range downloader**. It modifies the `eventLoop` within `grpc_reader_multi_range.go` by introducing an internal request queue. This crucial change **decouples request sending from command processing**, effectively preventing deadlocks that previously arose during channel coordination. The fix ensures more robust and concurrent operation of the storage layer's data retrieval mechanisms.

This commit initiates a **maintenance release** for the **Google Cloud Storage client library**, updating it to version `1.59.1`. This release primarily delivers **bug fixes** addressing issues within the `storage` module, specifically related to mandatory attributes in responses, ensuring the `attrsReady` channel is correctly closed when metadata is missing, and preventing unintended updates to global object read handles. By resolving these issues, the commit enhances the **stability and correctness** of the `storage` client, particularly for operations involving object metadata and reading.

This commit **adds a new integration test** to the **`storage`** subsystem. The test specifically verifies the correct functionality of the **`MultiRangeDownloader`** component when it is initialized with a read handle. This **enhances test coverage** for a critical data retrieval mechanism, ensuring its reliability and proper behavior in scenarios involving direct read access.

This commit performs a **maintenance release initialization** for the **`storage` client library**, updating its version from `1.58.0` to **`1.59.0`**. This **chore** incorporates **new features**, specifically adding default checksums for appendable writers, and includes **bug fixes** that refactor the `MultiRangeDownloader` to resolve deadlock and race conditions. The update affects the **`storage` module**'s internal versioning, snippet metadata, and librarian state, ensuring users receive the latest improvements and stability enhancements.

This commit **improves and adds tests** for the **`storage` module's `MultiRangeDownloader`**. It significantly enhances the `TestIntegration_ReadSameFileConcurrentlyUsingMultiRangeDownloader` in `storage/integration_test.go` by enabling it, increasing object size, and improving concurrency control and range verification. Additionally, new test cases are introduced in `storage/retry_conformance_test.go` to specifically verify `MultiRangeDownloader`'s handling of concurrent request recovery, invalid ranges, and download resumption. This work ensures the robustness and correctness of the multi-range download functionality under various concurrent and error conditions.