Navin Agarwal

This commit **implements new layer compatibility policies** across various Fluid Framework components, defining specific support windows between layers like Driver, Loader, Runtime, and DataStore. It introduces a central `LayerCompatibilityPolicyWindowMonths` constant within the **`client-utils`** package, which is then integrated into the compatibility validation logic of the **`loader`**, **`runtime`**, and **`datastore`** modules. This **new capability** automates the enforcement of these policies, ensuring that components adhere to defined generation compatibility requirements. The change **streamlines future policy updates** by centralizing configuration and includes **updated documentation** in `LayerCompatibility.md` to reflect these new guidelines.

This commit **refactors** and **updates** the **test compatibility configuration** to improve clarity and reflect current saturation data. It renames "LTS" (Long Term Support) references to "OCV" (Oldest Compatible Version) within `packages/test/test-version-utils/src/compatConfig.ts` and the `tools/pipelines/test-real-service.yml` pipeline, distinguishing test versions from actual LTS branches. As part of this **testing infrastructure update**, the designated oldest compatible version for testing is updated to `2.0.0-internal.5.4.2`. This change solely impacts the **test matrix** and does not alter the project's officially supported versions.

This commit **improves the diagnostic capabilities** for a specific **data corruption error** related to non-advancing `clientSequenceNumber` within the **container loader's delta management**. It modifies the `validateClientSequenceNumberConsistency` function in `packages/loader/container-loader` to now include the `clientSequenceNumber` in telemetry logs, providing crucial context for debugging this critical issue. Concurrently, this work **refactors** the **telemetry utilities** by introducing a new static `create` method for `DataCorruptionError` in `packages/utils/telemetry-utils`, which standardizes the capture of important message properties and enhances the overall **error reporting infrastructure**. This change is a **maintenance improvement** that directly impacts the ability to troubleshoot data integrity problems more effectively.

This commit **enhances Fluid Framework documentation** by introducing new `.md` files that comprehensively explain **compatibility types** and **layer compatibility**. It provides detailed guidance on **API, layer, cross-client, and data-at-rest compatibility** in `FluidCompatibilityConsiderations.md`, alongside an in-depth exploration of **Fluid architectural layers** in `LayerCompatibility.md` and a dedicated `LayerCompatibilityDevGuide.md`. This **documentation initiative** aims to significantly **improve understanding** for both **customers and developers** on how to effectively leverage and maintain compatibility within the Fluid ecosystem, particularly for the encapsulated model and feature development.

This commit introduces a **critical validation** within the **Delta Manager** to ensure that `clientSequenceNumber` values are always strictly increasing for each client when receiving operations. This **bug fix** addresses a reliability issue where duplicate sequence numbers, potentially from the service, would previously cause delayed and obscure failures, such as `assert 0x645` in the ID compressor. By adding immediate checks via `validateClientSequenceNumberConsistency` in `deltaManager.ts`, the system can now detect **data corruption** much earlier, providing clearer error messages and improving overall stability. New **test cases** were added to thoroughly cover this validation logic, ensuring robust handling of various sequence number scenarios.

This commit **reverts a previous change** that incorrectly removed the `testWorkspace` pipeline variable from the CI/CD configuration. This **bug fix** is crucial for the **CI/CD pipelines**, specifically restoring the functionality of the **end-to-end and stress test pipelines** which depend on this variable. By re-adding `testWorkspace` to the `tools/pipelines/templates/include-test-real-service.yml` template, it resolves a critical breakage in the automated testing infrastructure. This ensures the continued execution and reliability of these important test suites.

This commit **refines the layer compatibility validation logic** to only enforce checks on layers that explicitly expose their compatibility state, preventing erroneous failures for custom implementations like placeholder data stores. It introduces **new safety mechanisms** for the initial rollout, including a global feature flag (`allowIncompatibleLayersKey`) to disable all layer compatibility validation and another (`disableStrictLoaderLayerCompatibilityCheckKey`) to relax strict validation for the Loader layer. When validation is bypassed, a `LayerIncompatibilityDetectedButBypassed` telemetry event is logged. This work primarily impacts the **loader**, **container-runtime**, and **datastore** packages, along with **telemetry utilities**, by making the compatibility system more robust and configurable.

This commit **refactors** the Fluid compatibility layer system to enhance consistency and maintainability. It centralizes the `generation` value, critical for layer compatibility, by auto-generating it within the **`client-utils`** package. Consequently, all **Fluid compat layers** across various packages, including `local-driver`, `odsp-driver`, and `container-loader`, are updated to import this unified `generation` constant instead of using hardcoded values. This change ensures that all components consistently adhere to the same compatibility standard, simplifying future updates and improving the overall robustness of **Fluid's forward and backward compatibility**. The `generation` constant is also marked as `@internal` for clarity.

This commit **upgrades the API stability level** of the `layerIncompatibilityError` error type and its corresponding `ILayerIncompatibilityError` interface from legacy alpha to **legacy beta status**. This involves **refactoring** the core error definitions by moving `layerIncompatibilityError` from `FluidErrorTypesAlpha` to `FluidErrorTypes` within the `packages/common/core-interfaces` package. The change impacts the **error handling system** across various Fluid Framework packages, including `container-definitions`, `driver-definitions`, and `telemetry-utils`, ensuring all references and API reports are updated to reflect the new beta status. This promotion signifies increased confidence in the error's definition, providing a more stable API for consumers dealing with **layer incompatibility issues**.

This commit **increments the layer compatibility generation** for various **Fluid layers** from 2 to 3. It updates the compatibility state in the `local-driver`, `odsp-driver`, `routerlicious-driver`, `container-loader`, `container-runtime`, and `datastore` packages. This is a **routine maintenance update** to ensure ongoing compatibility across Fluid components, aligning with the expected monthly update cycle. The change represents a **manual compatibility bump** to maintain system stability and interoperability.

This commit performs **maintenance** by updating the `package.json` scripts within the **client-utils** package. It renames the command invocation from `flub layerCompatGeneration` to `compatLayerGeneration` to align with a recent change in the underlying command's name. This **refactoring** ensures that the build and compatibility generation processes for **client-utils** continue to function correctly, preventing broken scripts due to the command rename. The change is a direct follow-up to a previous standardization of command naming across the project.

This commit **introduces new format versions** (`ForestFormatVersion.v2` and `ForestSummaryFormatVersion.v3`) within the **`packages/dds/tree`** package to **enable incremental summaries** for the forest data structure. `ForestFormatVersion.v2` supports incremental encoding of forest data, while `ForestSummaryFormatVersion.v3` specifically allows incremental summaries and standardizes the top-level forest content key to `"contents"`. This **feature enhancement** updates the **`ForestSummarizer`** to leverage these new formats, requiring a `minVersionForCollab` of 2.74.0 to ensure all participating clients can correctly process these **performance-improving** summary changes. Validation was also added to ensure correct content key usage during summarizer loading.

This commit **introduces versioning to the Shared Tree's summary format** and its various summarizables, including the **Forest Summarizer**, **Schema Summarizer**, **Edit Manager Summarizer**, and **Detached Field Index Summarizer**. This **new capability** adds a metadata blob containing a format version to summaries, ensuring that clients encountering unrecognized summary formats will **fail fast** instead of silently breaking. By establishing a versioning framework via the new `VersionedSummarizer` abstract class, this change significantly improves the **robustness and forward compatibility** of the Shared Tree. This allows for safer evolution of its internal data structures by explicitly handling future summary format changes, starting with version 2 which is backward compatible with version 1.

This commit **enhances the test infrastructure** to support **compatibility testing** for packages introduced in later versions of the Fluid Framework, such as `@fluidframework/tree`. Previously, the test system only installed packages available across *all* supported older versions, limiting compat coverage for newer components. This **new capability** introduces a `minVersion` property for packages in `testApi.ts`, ensuring they are only installed when the target version meets this requirement. Consequently, the `CompatApis` now gracefully handle `undefined` entries for unavailable packages, and a new `mode` property is added to `CompatApis` to inform tests about the current compatibility configuration, enabling tests like `treeCompat.spec.ts` to conditionally run or skip based on version availability. This change significantly expands the scope of **compat testing**, allowing for more robust validation of newer Fluid Framework features.

This commit introduces a **new capability** to the **Fluid Framework test infrastructure**, enabling **compatibility testing** for packages that were not available in older versions. It enhances the `packages/test/test-version-utils` module by adding a `minVersion` property to package definitions in `testApi.ts`, ensuring packages are only installed and exposed in `CompatApis` when the target version meets this minimum requirement. This allows for more accurate **compat coverage** for features like **SharedTree** (`@fluidframework/tree`), which can now be tested against versions 2.0.0+ while gracefully skipping older, incompatible configurations, as demonstrated in `treeCompat.spec.ts`. Additionally, a `mode` property is added to `CompatApis` to help tests determine the current compatibility context and skip irrelevant scenarios, improving the robustness and accuracy of the **end-to-end test suites**.

This commit **fixes a bug** in the **Fluid Framework's DDS Tree** where **incremental summary and encoding** were not correctly applied to **map and record nodes**. Previously, the `getNodeEncoder` function within the **chunked forest codec** only checked for incremental encoding opt-in for object nodes, overlooking map and record types. The fix modifies `getNodeEncoder` in `packages/dds/tree/src/feature-libraries/chunked-forest/codec/schemaBasedEncode.ts` to properly recognize and apply the `IncrementalEncodingPolicy` to these node kinds, ensuring they are incrementally encoded when opted in. Additionally, the `IncrementalEncodingPolicy` type was **refactored** to allow an optional `fieldKey`, and **comprehensive tests** were added to validate incremental summarization across all node types, including object, map, array, record, and leaf nodes. This improves the efficiency and correctness of data synchronization for complex data structures within the **`@fluidframework/tree` DDS**.

This commit **refactors** the **`build-cli` package** by renaming the command `layerCompatGeneration` to `compatLayerGeneration`. This change improves clarity, explicitly indicating that the command is designed for generating compatibility layers rather than other layer types. The core command file `build-tools/packages/build-cli/src/commands/generate/compatLayerGeneration.ts`, its associated test file, and the `generate.md` documentation have all been updated to reflect this new naming convention. This **maintenance** change enhances the understandability and consistency of the build tools' command-line interface.

This commit **introduces and tags new `@alpha` APIs** to enable **incremental summarization** within the **SharedTree** DDS. It provides developers with `configuredSharedTreeAlpha`, `SharedTreeOptionsAlpha`, and related utilities like `IncrementalEncodingPolicy` and `incrementalEncodingPolicyForAllowedTypes` to **opt-in specific data types for more efficient summary generation**. Concurrently, significant **refactoring** consolidates internal tree compression strategies, streamlining the `TreeCompressionStrategy` and removing deprecated variants across various `dds/tree` components. This **new capability** allows applications to improve performance by reducing summary size and generation time for `SharedTree` instances by leveraging partial summaries.

This commit introduces a **new capability** to the **`build-tools` release process** by adding a mandatory check for **layer compatibility generation**. It integrates a new `checkCompatLayerGeneration` function into the `build-cli`'s release state handler and state machine, specifically within `build-tools/packages/build-cli/src/handlers/checkFunctions.ts`. This function ensures that for minor and patch releases, the layer compatibility metadata is up-to-date by running `pnpm run -r layerGeneration:gen` and prompting for commits if changes are detected. This enhancement improves the **robustness and reliability of Fluid Framework releases** by preventing outdated compatibility information from being published, similar to existing checks for type tests and release notes.

This commit **integrates automated layer compatibility generation** for the `client-utils` package, leveraging a recently introduced command. It adds a `layerGeneration:gen` script to `client-utils`'s `package.json` and updates `fluidBuild.config.cjs` to execute this command, which then auto-generates the `layerCompatMetadata` and a new `layerGenerationState.ts` file. This **new capability** within the **build system** and `client-utils` module **automates the management of layer compatibility metadata**, setting the generation value to 2. The change streamlines **maintenance** for `client-utils` by automating a previously manual process and provides a template for other packages to adopt similar automation.