Kathy

This commit performs a **documentation update** and **maintenance task** to reflect the upgrade of the **`@cloudflare/playwright` package to version 1.2.0**. It updates the version references in the `src/content/docs/browser-rendering/playwright/index.mdx` file, ensuring the documentation accurately states the new underlying Playwright version as v1.58.2. Concurrently, a new release note entry is added to `src/content/release-notes/browser-rendering.yaml` to formally announce this update. This ensures that users consulting the **browser rendering documentation** and **release notes** have access to the most current and accurate version information for the Playwright integration.

This commit **enhances the documentation** for the **`/crawl` REST API endpoint** by adding crucial details about how crawl results handle errors and content signals. It introduces a new subsection explaining how HTTP errors (e.g., 402, 403) are surfaced in `metadata.status` and `metadata.html`, and documents the new `crawlPurposes` optional parameter. Furthermore, it clarifies the behavior of "Content Signals" with categories like `search`, `ai-input`, and `ai-train`, including enforcement and how to narrow declared purposes, alongside a new troubleshooting entry for 400 Bad Request errors. This **documentation update** significantly improves user understanding of the endpoint's error reporting and content policy features within the `browser-rendering` module.

This commit **updates and refactors the Browser Rendering documentation** to enhance clarity regarding bot detection IDs and `robots.txt` configurations. It introduces a **new bot detection ID** for REST API and Workers Bindings, while clarifying that crawl endpoints utilize an existing ID, consolidating all such IDs into a single, canonical source within the `automatic-request-headers.mdx` page. This **documentation improvement and maintenance** effort ensures users have accurate and easily accessible information for configuring bot detection and understanding crawl-specific blocking, impacting how users manage automated requests and search engine interactions. The `faq.mdx` and `robots-txt.mdx` pages were also updated for better readability and to reference these consolidated IDs.

This commit performs **documentation maintenance** by **fixing outdated information** within the **Queues tutorial** specifically covering **web crawling with browser rendering**. It updates the `src/content/docs/queues/tutorials/web-crawler-with-browser-rendering/index.mdx` file to correct details related to browser rendering limits. This ensures that users following the tutorial receive accurate and current guidance, improving the overall clarity and relevance of the **documentation** for this feature.

This commit **updates the documentation** for the **Browser Rendering API**, specifically clarifying the **Workers AI** integration. It adds the default model name (`@cf/meta/llama-3.3-70b-instruct-fp8-fast`) to the `/json` endpoint page, along with a link to model documentation and guidance on using custom models. Additionally, it simplifies the intro paragraph of the `/json` endpoint docs and adds a cross-reference to these docs within the `/crawl` endpoint's product catalog extraction section. This **documentation update** improves user understanding of the default AI model and provides clearer guidance for developers interacting with the **Browser Rendering API**.

This commit **updates documentation** to accurately reflect the behavior of the **`/crawl` endpoint** within the browser rendering REST API. It **clarifies** that the `User-Agent` for requests made via the `/crawl` endpoint is **fixed and not customizable**, removing a partial that previously suggested otherwise. The changes affect `crawl-endpoint.mdx` by adding a dedicated `User-Agent` subsection and `automatic-request-headers.mdx` by noting `/crawl` as an exception to general `User-Agent` customization. This **documentation update** ensures users have correct expectations regarding `User-Agent` control for this specific API, preventing confusion or failed customization attempts.

This commit **enhances documentation** by adding crucial guidance to the `setting-custom-user-agent` partial. This update advises destination servers to leverage **non-configurable headers** for identifying or blocking Browser Rendering requests, providing a clearer method for traffic management. It includes a link to the automatic request headers reference and propagates this important information across all **REST API endpoint pages**, such as content, screenshot, and PDF, ensuring consistent user guidance. This is a **documentation update** to improve user understanding of custom user agent behavior.

This commit significantly **enhances the Browser Rendering documentation** by adding a new, detailed section on **blocking crawlers with `robots.txt`** in `robots-txt.mdx`, complete with practical examples for various blocking scenarios. It also **improves documentation discoverability** by adding a cross-reference from `crawl-endpoint.mdx` and updating related resources. Furthermore, this commit **fixes several broken internal anchor links** within `automatic-request-headers.mdx` and `robots-txt.mdx`, ensuring accurate navigation and a better user experience for readers. This work provides clearer guidance for users managing crawler access and improves the overall quality of the documentation.

This commit significantly **improves the documentation** for the **Browser Rendering REST API**, addressing feedback related to API token permissions and bot protection. A new shared partial, `rest-api-token-permissions.mdx`, is introduced and integrated across all Browser Rendering REST API endpoint documentation, providing consistent and clear guidance on required API token permissions. Specifically, the `crawl-endpoint.mdx` documentation receives enhanced details on bot protection cautions and a new FAQ link, complemented by clarifications in the general `faq.mdx` regarding bot protection and allowlisting. This **documentation maintenance** effort ensures users have more comprehensive and user-friendly information for interacting with the Browser Rendering service.

This commit **enhances documentation** by providing comprehensive details regarding `User-Agent` headers across the system. It **updates the browser rendering reference documentation** (`automatic-request-headers.mdx`) to include a new section on `User-Agent` defaults and customizability, while also clarifying non-configurable headers. Additionally, the **REST API documentation for the crawl endpoint** (`crawl-endpoint.mdx`) is updated to specify its default `User-Agent` and improve `robots.txt` troubleshooting guidance. This **documentation update** ensures users have clearer information on how `User-Agent` headers are managed, improving clarity for developers and users interacting with these features.

This commit **adds and refines a changelog entry** for the new `/crawl` endpoint within the **Browser Rendering API**. It provides a clearer description of the endpoint's functionality, highlighting its use cases for training models and monitoring content. This **documentation update** ensures users have a comprehensive understanding of this new capability, including an updated `curl` command example for easier adoption. The work primarily affects the **changelog documentation** for the Browser Rendering service, improving user clarity and feature discoverability.

This commit **restores** the documentation for the new **Browser Rendering `crawl` REST API endpoint**, reversing a previous double revert. It reintroduces comprehensive documentation for the `/crawl` endpoint, detailing its usage, parameters, and specific limits within the **Browser Rendering** section of the project's documentation. This **documentation restoration** also includes a release note entry announcing the new capability and adds a troubleshooting section for JSON extraction to the existing `/json` endpoint documentation. The change ensures that users have complete and up-to-date information for this re-enabled API feature.

This commit delivers substantial **documentation updates** for the **browser rendering REST API**, specifically enhancing guidance on capturing authenticated page screenshots. A **new FAQ entry** has been added to `faq.mdx`, comprehensively explaining how to render authenticated pages using various REST API authentication methods. Additionally, the `screenshot-endpoint.mdx` documentation now includes **expanded examples** for both **HTTP Basic Auth** and **Token-based authentication**, with `examples.mdx` updated to reference these new methods. This **documentation enhancement** significantly improves developer understanding and usability when integrating authenticated page capturing via the REST API.

This commit performs a **documentation improvement** by replacing a blurry screenshot within the **Browser Rendering changelog**. It specifically targets the `2026-03-04-br-rest-api-limit-increase.mdx` entry, updating the image rendering from an Astro Image component to standard Markdown syntax. This **maintenance** task ensures that users viewing the changelog will encounter a clearer, higher-quality visual aid. The change enhances the overall readability and professionalism of the documentation for the **Browser Rendering REST API limit increase** feature.

This commit introduces a **documentation update** by adding a new changelog entry for the **Browser Rendering** service. The entry specifically details an **increase in the REST API rate limit**, providing crucial information for users. This **maintenance** task ensures that customers on **Workers Paid plans** are informed about the enhanced capabilities and higher usage thresholds now available for the **Browser Rendering REST API**.

This commit **refactors** the **browser rendering documentation** by extracting the "Integration methods" content into a new reusable partial, `src/content/partials/browser-rendering/integration-methods.mdx`. This **documentation refactoring** moves the integration methods below use cases for improved logical flow within the guides. Both the `src/content/docs/browser-rendering/get-started.mdx` and `src/content/docs/browser-rendering/index.mdx` pages are updated to consume this new partial, eliminating content duplication. The change significantly enhances the organization and maintainability of the **browser rendering guides**, providing a more consistent experience for users.

This commit **increases the REST API rate limit** for the **Browser Rendering** service from 180 requests per minute to 600 requests per minute (10 per second). This **policy change** significantly enhances the capacity for users interacting with the service programmatically. The update includes modifying the `src/content/docs/browser-rendering/limits.mdx` documentation to reflect the new limits and adding a corresponding entry to the `src/content/release-notes/browser-rendering.yaml` file. This change directly benefits users by allowing for higher throughput when utilizing the **Browser Rendering REST API**.

This commit **updates the documentation** for the `/json` endpoint by **removing a specific caution note** related to **Anthropic models**. This **documentation maintenance** task reflects an updated status or resolution of previous concerns, indicating that the caution is no longer necessary for these models when used with the `/json` endpoint. Consequently, users consulting the documentation will find more current and accurate information regarding the capabilities and stability of Anthropic models within this specific API context.

This commit primarily **updates the documentation** by **adding a new release note** to `src/content/release-notes/browser-rendering.yaml`. This release note highlights a **new tutorial** focused on generating Open Graph (OG) images for Astro sites, enhancing the project's educational resources. Additionally, it includes minor **maintenance updates** to the **sidebar navigation order** for the `examples`, `get-started`, and `rest-api` documentation pages within the `browser-rendering` section. This ensures the documentation remains current and well-organized for users exploring browser rendering capabilities and new Astro-related features.

This commit **introduces comprehensive documentation** for configuring `robots.txt` and sitemaps within the **Browser Rendering** product. It adds a new reference page, `src/content/docs/browser-rendering/reference/robots-txt.mdx`, providing best practices and guidance for managing search engine crawling and indexing. This **documentation update** also includes a corresponding entry in the `src/content/release-notes/browser-rendering.yaml` to inform users about the new resource. The change aims to enhance user understanding and control over search engine visibility for their Browser Rendering applications.