David Wei

This commit performs a **performance refactoring** of the **`IoUringZeroCopyBufferPool`** by **removing the synchronous flush mechanism**, which was causing excessive overhead. It also **optimizes buffer management** within `IoUringZeroCopyBufferPoolImpl` by replacing `std::queue` with a more efficient `FixedCapacityRingQueue` for pending buffers. Furthermore, the buffer refill logic is updated in methods like `refillBuffer` and `returnBuffer` to directly return incoming buffers into the refill queue, avoiding an extra queue operation. This **optimization** aims to **improve the efficiency and reduce overhead** of `io_uring` zero-copy operations. The removal of the sync flush is a temporary measure while a more robust solution is designed.

This commit introduces a **new capability** to the `folly/container` library by adding the **`FixedCapacityRingQueue`** class, a highly efficient, fixed-capacity ring buffer queue backed by a power-of-two sized array. This new data structure is intended to optimize buffer management, specifically designed to replace `std::queue` as the `pendingQueue_` within the `IoUringZeroCopyBufferPool` for holding buffers that userspace cannot immediately place into the refill queue. The addition includes comprehensive unit tests to ensure the reliability and correctness of the `FixedCapacityRingQueue` implementation. This enhancement provides a more performant and predictable queue mechanism for critical buffer handling operations.

This commit introduces **new capabilities** and a **security enhancement** to the **`ClientFactory`** within the `thrift/conformance/stresstest/client` module. It adds comprehensive support for **I/O Uring socket options**, leveraging new functions like `getIoUringSocketOptions` and `createSocketWithIOUring` to improve socket performance. Concurrently, it implements crucial logic to **stop TLSv1 connections**, thereby enhancing FIZZ security for client-side communication. This change primarily affects the stress test client's connection handling, providing both performance optimizations and a critical security upgrade by disabling an outdated TLS protocol.

This commit introduces a **new capability** by adding support for `io_uring` and `StopTLSv2` within the **Thrift client factory**. Specifically, the `createIOUringFizz` function in `thrift/conformance/stresstest/client/ClientFactory.cpp` is updated to configure `ThriftParametersClientExtension` and `FizzThriftFactory` to enable `StopTLSv2`. This enhancement allows `io_uring`-based Thrift clients to utilize the `StopTLSv2` protocol, expanding their secure communication options. The changes were modeled after existing `epoll` implementations to ensure consistency.

This commit delivers a **bug fix** for the **Thrift conformance stresstest client**, specifically addressing an issue within the `ClientFactory`'s handling of `StopTLSv1` connections. Previously, the `FizzStopTLSConnector` failed to invoke connect success callbacks when creating sockets for `StopTLSv1` negotiation, which was critical for the test's connection synchronization latch. The fix ensures that the `createSocketWithEPoll` function now correctly triggers these callbacks, allowing the **stresstest's connection tracking mechanism** to function as intended and preventing potential test hangs or incorrect results.

This commit **enables native `io_uring` support for `AsyncSocket`** within the **Thrift conformance stresstest framework**. It introduces a new configuration flag, `io_uring_async_socket`, which allows the stresstest client and server to conditionally leverage `io_uring` for I/O operations. This **new capability** facilitates testing and validation of `AsyncSocket`'s `io_uring` integration, particularly to address issues like client-side zero-copy receive. By modifying socket creation logic and `io_uring` utility functions, this change provides a controlled environment to assess the performance benefits of `io_uring` for Thrift applications.

This commit introduces a **new capability** to the **Thrift server's peeking subsystem**, enabling support for **movable buffers**. It **enhances** the `peekSuccess` method within `thrift/lib/cpp2/server/peeking/PeekingManager.h` to allow pre-received data to be efficiently moved from the peeker to the underlying socket. This is a critical **enhancement** for integrating native `io_uring` support into `AsyncSocket`, as it allows `ReadCallbacks` to handle movable `IOBufs` and provide necessary read size hints. Consequently, this change facilitates optimized read operations, leveraging `io_uring`'s multishot `recv` for larger data transfers while gracefully falling back to fixed-length `recv` for smaller reads.

This commit **integrates** the **`liburing` third-party library** into the project, specifically importing its code from the master branch into the `third-party/liburing` directory. This **maintenance** action establishes a clean, pristine copy of the library's source, making its functionalities available for use by other project components. The import provides a foundational dependency, enabling future development that leverages `liburing`'s capabilities.

This commit **refactors** the `connect()` method across the **`AsyncSocket` family of classes**, including `AsyncSocket`, `AsyncSSLSocket`, and `AsyncIoUringSocket`, to improve clarity and type safety. It introduces a `std::variant` named `BindOptions` to encapsulate the mutually exclusive `bindAddr` and `boundFd` parameters, making it explicit that only one binding option can be provided. This **API change** enhances the robustness of the **network connection binding mechanism** within Folly's asynchronous I/O library. The change affects the method signatures in `AsyncSocketTransport.h` and its implementations, requiring updates to all call sites and derived classes, as reflected in the updated test suites.

This commit **refactors** the **`AsyncSocketTransport`** component to enhance code clarity and type safety within its connection mechanism. Specifically, the `connect()` method now utilizes a `std::variant` to explicitly manage the mutually exclusive `bindAddr` and `boundFd` parameters, making their intended usage clearer. This internal design improvement simplifies the API for consumers, as demonstrated by the updated `createIOUring` and `createIOUringTLS` functions in the `ClientFactory` of the `thrift/conformance/stresstest` client. These functions now leverage `AsyncSocketTransport::BindOptions` for socket binding, leading to more streamlined `connect` method calls and improved maintainability of the socket transport layer.

This commit introduces a **new capability** to the **Thrift conformance stress test** by making the `io_uring` zero-copy arena size configurable. Previously a hardcoded value, this critical parameter for `io_uring` performance can now be specified via command-line flags. The change primarily affects the `stresstest` client configuration, `ClientFactory` for socket creation, and `IoUringUtil` for flag definition. This **feature enhancement** provides users with greater control over resource allocation for `io_uring` operations, allowing for more flexible and optimized stress testing scenarios.

This commit provides a **bug fix** for the **Thrift Cpp2 server** that ensures **zero-copy transmission** is correctly utilized for plaintext connections. Previously, when an `AsyncIoUringSocket` was created from an existing `AsyncSocket` within `Cpp2Worker::onNewConnectionThatMayThrow`, the zero-copy configuration was not propagated, leading to inefficient data transfer. The fix explicitly re-applies the zero-copy enable function to the `AsyncIoUringSocket`, thereby restoring the intended performance benefits for **plaintext AsyncIoUringSockets** when zero-copy is enabled. This change impacts the `Cpp2Worker`'s connection handling logic, ensuring optimal network performance for supported connections.

This commit introduces a **new feature** by **enabling io_uring zero-copy transactions** on the **server side** of the **Thrift conformance stress test framework**. The `createStressTestServer` function in `thrift/conformance/stresstest/server/StressTestServer.cpp` is updated to leverage this high-performance I/O mechanism. This enhancement allows for more efficient data transfer during stress testing and includes the optional initialization of an `IoUringArena`. The change primarily impacts the **performance testing capabilities** of the Thrift framework, allowing the stress test server to utilize advanced kernel features for I/O efficiency.

This commit introduces a **new capability** for `folly::IOBuf` memory management by defining `folly::IOBufFactory`, a flexible function type for creating `unique_ptr<folly::IOBuf>` instances. It also adds `makeIOBufArenaFactory`, a helper function in `folly/memory` that generates an `IOBufFactory` utilizing templated arena allocators, such as `IoUringArena` or `JemallocHugePageAllocator`. This **infrastructure improvement** aims to standardize and optimize `IOBuf` allocations, moving away from direct `IOBuf::create` calls. The new factory mechanism is intended for adoption by core components like `IOBufQueue` and various parts of **Thrift**, enabling more efficient memory handling, especially in high-performance contexts.

This commit introduces a **new capability** to `folly::IOBufQueue`, enabling it to use a custom `folly::IOBufFactory` for `IOBuf` allocations instead of the default `IOBuf::create`. This **enhances memory management flexibility** within the `folly::io` subsystem by providing a `setIOBufFactory` method to configure a custom factory. The implementation carefully preserves the `IOBufQueue`'s 64-byte size through internal data packing, ensuring minimal impact on existing layouts while enabling users to optimize `IOBuf` creation for their specific needs. This change primarily affects how `IOBuf`s are allocated during operations like `append`, `preallocateSlow`, and `split` within the queue.

This commit **refactors** the **Thrift Rocket transport payload serialization** mechanism to introduce an `IOBufFactory` for buffer allocation. An `IOBufFactory` is now passed into `SerializedRequest` and subsequently plumbed through `RocketClientChannelBase::sendThriftRequest` to various `PayloadSerializerStrategy` implementations, including `DefaultPayloadSerializerStrategy`, `ChecksumPayloadSerializerStrategy`, and `CustomCompressionPayloadSerializerStrategy`. This **enhancement** allows for **custom buffer allocation strategies** during payload packing, particularly impacting methods like `makePayloadWithoutHeadroom`. The change provides greater control and flexibility over memory management for outgoing Thrift requests.

This commit introduces a **new feature** to the **Thrift C++2 asynchronous client library** by integrating an `IOBufFactory` into the `GeneratedAsyncClient`. This factory, configured via the `Options` struct and managed by new `setIOBufFactory` and `getIOBufFactory` methods, will now be responsible for *all* `IOBuf` allocations within the client. This enhancement centralizes `IOBuf` creation, allowing for custom allocation strategies and potentially optimizing memory management for `GeneratedAsyncClient` instances.

This commit **introduces an `IOBufFactory`** into the **Thrift RocketClient** to enable custom buffer allocation during serialization. It **adds a new capability** by plumbing the `IOBufFactory` through `RocketClientChannelBase::sendThriftRequest`, `RocketClient`, and `RequestContext` down to the `Serializer` and various `Frames` serialization methods. This allows for more flexible and potentially optimized memory management within the **Thrift Rocket transport** by providing a mechanism to control `IOBuf` allocation. The change impacts the core serialization path for all Rocket frames, ensuring the factory is used consistently.

This commit introduces a **new integration test** for the **Rocket client transport** within Thrift, specifically targeting `IOBufFactory` integration. The test verifies that when a **Thrift client** is configured to use an `IOBufFactory`, the serialized frames are correctly allocated from that factory's memory. It utilizes an `IoUringArena` as the concrete `IOBufFactory` implementation to confirm proper buffer allocation and memory management. This **quality assurance** effort enhances confidence in the correct functioning of custom buffer allocation strategies for Thrift client serialization.

This commit **enhances the `RequestChannel`'s message serialization process** by plumbing an `IOBufFactory` through the `preprocessSendT` method. It modifies the **Thrift C++2 async client generation template** to pass this factory, which is then used for initial `IOBuf` allocation and set into the `IOBufQueue` for subsequent buffer needs. The `SerializedRequest` struct is updated to store this factory, enabling more efficient and controlled memory management for outgoing requests. This **optimization** aims to improve performance and reduce memory overhead in **Thrift C++2 applications** by centralizing `IOBuf` creation.