Shenyang Cai

This commit performs **maintenance** by **disabling a specific system test**, `test_read_csv_for_names`, within the **BigFrames testing suite**. The test, located in `packages/bigframes/tests/system/small/test_session.py`, is now skipped using a pytest marker. This temporary measure prevents CI failures related to this particular test, allowing other development to proceed while the underlying issue is investigated. The **BigFrames session functionality** related to CSV name reading will no longer be validated by this specific test in automated runs.

This commit **introduces a new example notebook** to the `bigframes` library, specifically `packages/bigframes/notebooks/generative_ai/ai_movie_poster.ipynb`. This **new capability** demonstrates how to leverage **BigQuery DataFrames AI functions** for analyzing movie posters, providing a practical, real-world application. The notebook is intended as a reference for an upcoming tech blog, enhancing the **documentation and user understanding** of generative AI features. A link to this new resource has also been integrated into the `packages/bigframes/docs/user_guide/index.rst` to improve discoverability.

This commit introduces a **new capability** for **BigFrames AI operations**, specifically `bf.ai.if_`, `bf.ai.classify`, and `bf.ai.score`. It enables these functions to be invoked **without explicitly providing a connection ID**, defaulting to the use of **End-User Credentials (EUC)** for authentication when no connection is specified. This change simplifies the API for users who wish to leverage their default credentials, significantly enhancing the **usability** of the AI functions. The update involves modifying the `AIIf`, `AIClassify`, and `AIScore` operation classes in both the `bigframes.operations.ai_ops` module and the vendored Ibis library to make the `connection_id` attribute optional, alongside adding comprehensive unit tests and fixing a minor typo in the SQLGLot compilation logic.

This commit **deprecates the `claude-3-5-sonnet` model** within the **`bigframes` LLM integration**, marking it as unsupported for future use. It includes **documentation updates** for the `Claude3TextGenerator` in `bigframes.ml.llm` to reflect this deprecation. Furthermore, an example notebook (`remote_function_vertex_claude_model.ipynb`) is **refactored** to migrate its usage from the deprecated model to `claude-3-haiku`, alongside the removal of outdated cells. Finally, **system tests** in `test_llm.py` are updated to exclusively test with `claude-3-haiku`, ensuring the codebase and examples align with current model availability. This **maintenance and cleanup** effort guides users towards supported LLM versions.

This commit introduces the **new `dt.tz_localize()` method** to the **`bigframes` library**, enabling users to localize timezone-naive datetime Series to UTC or remove existing timezone information. This **enhances datetime manipulation capabilities** by providing a crucial tool for handling timezones, with initial support for `None` and `"UTC"` time zones. The implementation involves defining the `tz_localize` interface in `DatetimeProperties` and its concrete method in `bigframes.operations.datetimes`, alongside **updates to the Ibis and SQLGLOT compilation backends**. These backend enhancements expand support for additional input types in `to_datetime` and `to_timestamp` operations, improving the overall robustness of datetime conversions within the **`bigframes` framework**.

This commit prepares for the **release of `bigframes` version 2.36.0**, a **maintenance** task initiated by the Librarian CLI. It updates the version numbers across the `bigframes` package, specifically in `packages/bigframes/bigframes/version.py` and `packages/bigframes/third_party/bigframes_vendored/version.py`. This new version introduces **initial support for BigLake Iceberg tables** and the `bigquery.ai.generate_table` function, alongside several **documentation improvements** for multimodal features. Merging this pull request will automatically trigger the official release, making these new capabilities and updated documentation available to users of the **bigframes** library.

This commit introduces the **new `bigquery.ai.generate_text` function** to the `bigframes` library, providing a **new capability** for text generation, and also **fixes a SQL syntax bug** affecting the `ai.generate_embedding` function. Significant **refactoring** was performed to improve code organization by centralizing common utility functions like `_to_sql` and `_get_model_name_and_session` into a new `_operations/utils.py` module. This refactoring enhances reusability and maintainability across **BigQuery ML and AI operations**, with comprehensive new unit and system tests added to ensure the **correctness and stability** of these updated functions.

This commit introduces a **new capability** to track and log data type usage in BigQuery jobs initiated by BigFrames. It modifies the **`bigframes.core.compile`** modules, specifically `ibis_compiler.py` and `sqlglot/compiler.py`, to encode type references from compiled SQL results and store them in the `CompileResult`. This encoded information is then attached as a `bigframes-dtypes` label to BigQuery job configurations by the **`bigframes.session.bq_caching_executor`**. A new system test in `test_session_logging.py` verifies this **usage logging** functionality, providing better insights into how data types are utilized within BigFrames workloads.

This commit introduces a **new capability** to the **`bigframes.core.logging`** module by adding functions to **traverse the BigFrames Expression Tree (BFET)** and **encode the data types used within it**. Specifically, new functions like `encode_type_refs` are added to `data_types.py` to extract and represent type usage from expressions, alongside an update to `_get_dtype_mask` to handle `None` types. This foundational work is crucial for **enhancing logging and potentially optimizing SQL dispatch** by enabling the inclusion of encoded type usage in `job_config.label`. Comprehensive unit and system tests have been added and updated to validate this new functionality.

This commit **deflakes** the doctest examples within the `generate` function's docstring in **BigFrames AI operations**. It updates the docstring in `packages/bigframes/bigframes/bigquery/_operations/ai.py` to **skip** these examples during automated testing and adjusts their expected output format. This **maintenance** change improves the reliability of the test suite by preventing intermittent failures. The core functionality of the AI generation feature remains unchanged, with the focus solely on enhancing test stability.

This commit **establishes foundational data type definitions and bit mask generation logic** for the **`bigframes.core.logging`** subsystem. It introduces a new module, `data_types.py`, which defines functions like `_get_dtype_mask` to convert various data types into specific bit masks, reserving the least significant bit for unknown types. This **chore** is a crucial preparatory step, providing the necessary data structures and conversion utilities for future logging enhancements. While it **does not yet implement the full logging functionality**, such as tree traversal, it lays the essential groundwork for improved type-aware logging within `bigframes`.

This commit performs a **refactoring** by **relocating the `log_adapter.py` module** from `packages/bigframes/bigframes/core/` into a new `packages/bigframes/bigframes/core/logging/` directory. This organizational change affects numerous internal modules within the **`bigframes` library**, including `bigquery`, `core`, `ml`, `operations`, `pandas`, `session`, and `streaming` components, as well as associated unit tests, all of which have had their import paths updated to reflect the new location. The primary purpose is to establish a dedicated `logging` subdirectory to better organize existing and future logging utilities, enhancing modularity and maintainability. This internal structural change has no impact on the public API or user-facing functionality.

This commit **introduces a new `weekday` property** to the **BigFrames `DatetimeMethods` operations**, enabling users to directly retrieve the day of the week from datetime objects. This **new feature** enhances the **datetime manipulation capabilities** within the `bigframes` package, aligning its API more closely with pandas-like functionality. The implementation is added to `bigframes/operations/datetimes.py`, accompanied by a new `test_dt_weekday` in `test_datetimes.py` to verify its correctness. A corresponding `weekday` property is also introduced in the vendored pandas `DatetimeProperties` accessor, initially raising a `NotImplementedError` as a placeholder for future integration.

This commit introduces a **new capability** for **BigQuery AI functions** within BigFrames, enabling them to utilize **end-user credentials** by default when a `connection_id` is not explicitly provided. The `bigframes.bigquery.ai` generation functions, including `generate` and `generate_bool`, are updated to conditionally pass the `connection_id` to the underlying SQL, allowing for more flexible credential management. This **feature enhancement** involves modifications to the SQL generation logic in `bigframes.core.compile.sqlglot.expressions.ai_ops` and updates to type hints in `bigframes.operations.ai_ops` to support optional `connection_id` values. This change simplifies the invocation of AI models by removing the implicit requirement for a connection ID in certain scenarios, improving user experience for AI operations.

This commit introduces a **bug fix** for the **`bigframes` library**, specifically addressing an issue where calling the `info()` method on **`bigframes.dataframe` objects** with no rows or no columns would lead to errors. The `info()` method has been updated to **correctly display information** for these edge cases, preventing crashes and improving the method's robustness. This ensures a more consistent and reliable user experience when inspecting DataFrame metadata, even for empty or structurally incomplete DataFrames. New **system tests** have been added to verify the `info()` method's behavior for DataFrames lacking rows, columns, or both.

This commit introduces a **new capability** by adding support for the `left_index` and `right_index` parameters to the `merge` operation within the **BigFrames library**. It enhances the `DataFrame.merge` and underlying `Block.merge` methods, specifically in `packages/bigframes/bigframes/core/blocks.py` and `packages/bigframes/bigframes/core/reshape/merge.py`, to allow merging on the index of the left or right DataFrame. This involves new logic for **column coalescing and index resolution**, providing users with more flexible data joining options. The change impacts the **`bigframes.dataframe`** and **`bigframes.core.reshape`** modules, and includes updates to vendored pandas `merge` signatures and comprehensive system tests.

This commit performs a significant **refactoring** of the **`bigframes`** library by **relocating the main `merge` logic** from the `DataFrame` class to a new, dedicated **`bigframes.core.reshape.merge`** package. The `DataFrame.merge` method in `bigframes/dataframe.py` is now a thin wrapper, delegating to this centralized implementation. This change improves code organization and modularity by separating concerns, making the `DataFrame` class leaner and the merge functionality more reusable and maintainable. Associated validation helpers, such as `_validate_left_right_on`, were also moved or removed from the original `dataframe.py` file as part of this cleanup.

This commit **refactors** the **BigFrames AI functionalities** by **migrating** the `ai.forecast` operation to the **new `bigframes.bigquery.ai` module**. It **introduces** a **new `forecast` function** within this module, enabling time series forecasting using the **TimesFM model**. Concurrently, the commit **deprecates** the direct `DataFrame.ai` accessor, now issuing a `FutureWarning` to guide users towards the consolidated `bigframes.bigquery.ai` module for all AI operations. This change streamlines the API for AI features and prepares for future enhancements.

This commit introduces a **new capability** to the **BigQuery dry run report** by including **local data bytes** in the total processed data when available. The `bigframes.session.dry_runs` module was enhanced with a new `get_local_bytes` function and modifications to `get_query_stats_with_dtypes` to accurately incorporate this local data into the overall byte count. This ensures that users receive a more comprehensive and accurate estimate of the total data processed, improving the utility of the **dry run report** for cost and performance analysis. The `_get_dry_run_stats` method in `bigframes.core.blocks` was updated to support this new data inclusion.

This commit introduces a **fix** to the **`google-cloud-bigquery` client library** by updating the `PathType` type hint definition. It now explicitly includes `io.IOBase`, allowing for a broader range of **file-like objects** to be correctly recognized and passed to methods expecting a path. This change improves the **type hint accuracy and flexibility** within `google.cloud.bigquery.client.py`, ensuring better compatibility for users providing various `io.IOBase`-derived objects. The update resolves potential type checking issues and enhances the robustness of the library's API.