Luke Sandberg

This commit introduces a **compatibility fix** for **Turbopack's `turbo-persistence` crate** by **disabling BMI2 instruction set usage** in its `qfilter` dependency. Specifically, it adds the `legacy_x86_64_support` feature to `qfilter` in `turbopack/crates/turbo-persistence/Cargo.toml`. This change is a **bug fix** that **enables support for older Intel CPUs** which lack BMI2 extensions, resolving a critical runtime issue.

This commit significantly **refactors** `turbo-tasks` by moving session dependency declarations from runtime calls to a compile-time `#[turbo_tasks::function(session_dependent)]` attribute, which enables eager aggregation number assignment for **improved performance** by preventing long dirty-propagation chains. It also **fixes** a critical issue in **`turbo-tasks-fetch`** by implementing **TTL support** based on HTTP `Cache-Control` headers, ensuring external resources are refreshed correctly and transient network errors are retried on session restore. Finally, the commit simplifies internal state management by replacing the `TransientState` mechanism with a localized `TransientCache` in `turbopack-ecmascript`, enhancing overall system reliability and API clarity.

This commit provides a **critical bug fix** for **Turbopack's CSS Hot Module Replacement (HMR)**, specifically addressing issues encountered on **Safari**. It resolves a regression where Safari's server-side cache-busting `?ts=` query parameter was incorrectly incorporated into HMR asset suffixes, causing style updates to fail due to a WebKit preload cache bug. The fix involves modifying `turbopack/crates/turbopack-ecmascript-runtime/js/src/browser/runtime/dom/dev-backend-dom.ts` to strip query strings when matching existing `<link>` elements and extending cache-busting logic to Safari, alongside an adjustment in `turbopack/crates/turbopack-ecmascript-runtime/js/src/browser/dev/hmr-client/hmr-client.ts` to ensure `total` HMR updates correctly supersede prior ones. This significantly improves the **developer experience** for users working with Turbopack on Safari by enabling reliable and instant CSS style updates.

This commit introduces a **performance optimization** within the **Turbopack JavaScript runtime** by replacing regular expression-based checks for file extensions. Specifically, the `isJs` and `isCss` utility functions in `turbopack/crates/turbopack-ecmascript-runtime/js/src/browser/runtime/base/runtime-base.ts` are **refactored** to use more efficient string operations like `indexOf` and `startsWith`. A new shared helper, `endsWithExtension`, is extracted to accurately determine file extensions without allocating substrings, significantly reducing overhead on a hot path and improving the overall efficiency of module path processing.

This commit introduces significant **performance optimizations** to the **`turbo-persistence`** crate, primarily targeting **CPU usage during compaction**. It achieves this through several **refactorings**: flattening the `StaticSortedFileIter` structure and optimizing key block parsing with direct byte indexing, introducing `RcBytes` for thread-local, non-atomic reference counting to reduce overhead during single-threaded SST iteration, and simplifying the `MergeIter::next` method for more efficient heap operations. These changes result in a **22-25% reduction in compaction CPU time** without regressing read path performance, enhancing the overall efficiency of the **Turbopack persistence layer**.

This commit **fixes a critical bug** in **Turbopack's handling of Node.js `worker_threads`**, specifically when `new Worker()` is called with the `{ eval: true }` option. Previously, **Turbopack** incorrectly attempted to resolve inline JavaScript code as a module reference, leading to build failures with libraries like jsPDF. The **Turbopack JavaScript parsing and reference resolution logic** within `turbopack-ecmascript` and `turbopack-core` has been updated to correctly identify and skip creating worker references for `eval: true` scenarios, while also adding warnings for dynamic `eval` options. This **bug fix** significantly **improves compatibility with dynamic worker creation patterns**, ensuring that projects using such libraries can build successfully with Turbopack. Comprehensive **test coverage** has been added to validate the fix.

This commit introduces a significant **refactoring** to **Turbopack's backend snapshotting and persistence logic**, primarily by switching from `restored` to `modified` flags to determine which task data categories are persisted. This change ensures that only truly **modified data is re-serialized**, avoiding redundant re-persistence of clean data and improving the efficiency of snapshot operations. It also **unifies the `process` and `process_snapshot` callbacks** within `take_snapshot` due to their now identical logic, simplifying the API. Furthermore, the handling of transient tasks in the persistence path is made more robust by changing silent returns to an `unreachable!` panic, preventing logical errors within the `turbo-tasks-backend` crate.

This commit **updates the `tokio` dependency** from `1.43.0` to `~1.47.3` in `Cargo.toml`, migrating to the `1.47.x` LTS line for extended support and stability. This **maintenance update** incorporates several **performance improvements** to the core runtime, a **soundness fix** for broadcast channels, and makes `select!` budget-aware for improved fairness. It also stabilizes new **runtime metrics** and introduces new **synchronization APIs** like `sync::SetOnce`. The change ensures the project benefits from the latest advancements, bug fixes, and extended support within the `tokio` ecosystem, with verified compatibility for breaking changes.

This commit implements a significant **performance optimization** within the **Turbopack `turbo-tasks-backend` crate**, specifically for **snapshot processing**. It **refactors** the storage mechanism to **share scratch buffers** across shards using **thread-local storage**, which drastically **reduces memory allocations** during task data encoding. This change primarily impacts `src/backend/storage.rs`, where functions like `take_snapshot` and `take_scratch_buffer` are modified, and also updates the `save_snapshot` API in `backing_storage.rs` and `kv_backing_storage.rs` to accept `IntoIterator` for improved flexibility. By buffering these resources, the commit aims to **enhance the efficiency** and **reduce the overhead** of snapshot operations.

This commit performs a **dependency upgrade** for `turbopack`, updating the `qfilter` crate to version `0.3.0-alpha.2` in `turbo-persistence/Cargo.toml`. This **performance improvement** significantly enhances the underlying quotient filter implementation used by the **`turbo-persistence`** module for **SST file lookups** and database operations. Benchmarks demonstrate substantial gains, including **22-34% faster lookups** and **10-17% faster DB-level reads**, particularly for cache misses. This upgrade results in a **more efficient and faster data access layer** across Turbopack's persistence mechanisms, with overall write paths improving by approximately 6%.

This commit delivers a **bug fix** and an **enhancement** to the **Turbopack build process** and its **tracing capabilities**. It resolves an issue where **Static Site Generation (SSG)** was unnecessarily blocked by Turbopack's internal persistence and cache-flush operations, allowing **SSG to run in parallel** with these tasks for improved build performance. Additionally, it **enhances build tracing** by including `turbopack-persistence` and `turbopack-compaction` spans in the `.next/trace-build` output, providing a more complete view of Turbopack's internal activities. This ensures that critical build events are no longer silently filtered, aiding in debugging and performance analysis of the **Turbopack build pipeline**.

This commit provides a **bug fix** to resolve the **consistent timeout of the `test-cargo-unit` CI job** within the **CI/CD pipeline**. It addresses this by **disabling Link-Time Optimization (LTO)** for unit test binaries, introducing a new `release-with-assertions-no-lto` Cargo profile in `Cargo.toml` and applying it to the `next-swc` test script. Additionally, the `turbo run` command for this job in `.github/workflows/build_and_test.yml` is updated to include a **remote cache timeout** and **streaming log output**, improving observability and preventing silent failures. This ensures **faster and more reliable CI feedback** for `next-swc` changes by preventing timeouts, without affecting LTO usage in production native builds.

This commit performs a significant **refactoring** within **Turbopack** to simplify the `ModuleReference` trait API and improve performance. It modifies `chunking_type()` and `binding_usage()` methods to return direct values instead of `Vc<T>` wrapped values, removing `#[turbo_tasks::function]` annotations and unnecessary async overhead. Additionally, the `EsmAssetReference::get_referenced_asset` task is removed, with its logic inlined into callers. This **performance optimization** across **Turbopack core**, **Next.js core**, **Turbopack CSS**, and **Turbopack Ecmascript** modules leads to a reduction of over 1.3 million total task invocations, 29 fewer task types, approximately 2 seconds faster build times, and a ~100MB decrease in max heap size.

This commit performs a significant **cleanup and refactoring** within the **`turbo-tasks-backend`** crate by **removing the unused LMDB database backend and the `ReadTransaction` abstraction layer**. This change simplifies the **persistence layer** by eliminating approximately 1,500 lines of dead code and removing `unsafe` blocks previously required for transaction lifetime management in `ExecuteContextImpl`. The `ReadTransaction` abstraction, which was only relevant for the removed LMDB backend, is now completely removed from core traits like `BackingStorageSealed` and `KeyValueDatabase`, as well as various `WriteBatch` implementations. This streamlines the database infrastructure's API, reduces complexity, and improves the overall maintainability and safety of the backend.

This commit performs a **cleanup** operation by **removing Git worktree configurations** that were inadvertently added to the repository in a prior commit. This **maintenance** task rectifies an accidental inclusion, ensuring the repository's integrity and preventing potential issues related to extraneous or misconfigured development artifacts. The change primarily affects the **repository's root or configuration files**, streamlining the project setup and removing unnecessary clutter.

This commit introduces a **performance optimization** within **Turbopack's persistence layer** by preventing unnecessary LZ4 compression attempts on small key blocks. Specifically, the `static_sorted_file_builder` in `turbo-persistence` now includes a minimum key size threshold, ensuring that the `flush_key_block` function only attempts compression when it's likely to be beneficial. This change addresses wasted CPU cycles, as current tables with small keys never meet the compression threshold, leading to a more efficient persistence process. The primary impact is a **reduction in CPU overhead** during Turbopack's internal data storage operations.

This commit **refactors** the **Turbopack error reporting mechanism** by moving the **code frame rendering logic from JavaScript to the native Rust NAPI bindings**. Previously, frame information was inefficiently passed between native and JS; now, the `crates/next-napi-bindings` crate directly implements `render_issue_code_frame` to generate these frames. This change **optimizes performance** by reducing interop overhead and simplifies the `formatIssue` function in `packages/next/src/shared/lib/turbopack/utils.ts`, which now consumes the pre-rendered `codeFrame`. The update ensures more efficient and consistent display of issue code frames within **Next.js applications** powered by Turbopack.

This commit introduces a **performance optimization** and **refactoring** to the **`turbopack` snapshotting mechanism** by moving the filtering of empty data shards. Previously, this filtering was a single-threaded, post-production step that could be expensive for the **`turbo-tasks-backend`**. The logic is now integrated into the `take_snapshot` function, allowing empty shards to be filtered **concurrently** during their production. This change reduces overhead and yields a **small but measurable improvement in build times**, specifically for filesystem cache write operations.

This commit introduces significant **CPU performance optimizations** within **Turbopack's persistence layer**, specifically for SST (Static Sorted Table) read operations. It **refactors** the hashing implementation in both **`turbo-persistence`** and **`turbo-tasks-hash`** by migrating from `twox-hash` to the more efficient `xxhash-rust` crate, eliminating heap allocations during hashing. Additionally, it optimizes byte ingestion for hash calculations in `KeyBase::hash` and enhances binary search algorithms within `static_sorted_file.rs` for faster index block lookups and duplicate key scans. These changes collectively reduce CPU overhead, leading to **faster Turbopack build times**, with benchmark results showing up to a 56% reduction in CPU usage for cached persistence reads.

This commit introduces a **performance optimization** and **refactoring** within the **Turbopack task persistence system** by eliminating the cloning of `TaskStorage` objects during database snapshotting. Instead of creating intermediate copies, the system now directly encodes `TaskStorage` references from the backend storage map, removing the need for `clone_meta_snapshot` and `clone_data_snapshot` operations. This change primarily affects the `turbo-tasks-backend` crate, including `backend/mod.rs` and `backend/storage.rs`, by streamlining the snapshot encoding process. By reducing CPU time and memory allocations, this **refactoring** improves efficiency during persistence, particularly benefiting CPU-constrained environments, even if overall wall-clock time remains similar for IO-bound operations.