Cameron Whiteside

This commit provides a significant **documentation update** for the **AI Crawl Control** feature, specifically detailing **WAF rule management** and **custom rule preservation**. It introduces new content explaining how to extend the AI Crawl Control `WAF` rule with custom modifications, moving relevant customization documentation to a dedicated configuration page. This **enhances user guidance** for advanced **WAF customization** and ensures users can effectively preserve their custom rules when integrating with AI Crawl Control. A corresponding changelog entry is also added to reflect these documentation improvements.

This commit significantly **expands and reorganizes the documentation for x402 agents**, transforming a single page into a dedicated `/agents/x402/` section. It introduces new, detailed guides on **charging for HTTP content and MCP tools**, as well as **paying with the Agents SDK and coding tools**. This **documentation improvement** clarifies the implementation of x402, removes redundant information, and fixes several code examples and structural issues. The refactored content provides a more comprehensive and user-friendly resource for developers working with x402-enabled agents, including specific instructions for custom Worker endpoints and simplified payment flows.

This commit significantly **enhances the documentation and analytics capabilities** for the **AI Crawl Control** feature. It introduces **new reference guides** for the **GraphQL API**, a comprehensive table of **bot detection IDs and user agents**, and a **Worker template** (`x402`) for payment-gated proxies. Additionally, the `analyze-ai-traffic` documentation is updated to reflect **new data transfer metrics** and path pattern grouping, while existing guides for **WAF** and **Transform Rules** are refined for clarity and troubleshooting. This work improves user understanding, integration possibilities, and the ability to monitor AI crawler interactions more effectively.

This commit **adds comprehensive documentation** for the newly introduced `AI Crawl Control Read Only` role. It updates the general **roles documentation** (`src/content/docs/fundamentals/manage-members/roles.mdx`) to include this specific role and creates a dedicated **changelog entry** (`src/content/changelog/ai-crawl-control/2026-01-13-ai-crawl-control-read-only-role.mdx`) to announce its availability. This is a **documentation update** that informs users about enhanced access control options within the **AI Crawl Control** feature. The change ensures that users are aware of the read-only capabilities for managing AI crawl settings, improving clarity on permission structures.

This commit **enhances the documentation** for the **AI Crawl Control** feature, specifically detailing the new Overview tab. It includes a new changelog entry for the tab and revises the `Analyze AI Traffic` and `Get Started` guides to instruct users on monitoring AI crawler activity via this new interface. Furthermore, the `AI audit` glossary is expanded with definitions for 'Operator' and 'Referrals', providing users with comprehensive resources to understand and utilize the updated **AI Crawl Control** capabilities.

This commit introduces **new customization features** for the **AI Crawl Control Pay Per Crawl** system, significantly enhancing its flexibility and transparency. It adds a **Discovery API** for AI crawlers to find payable content, enables **custom pricing** for site owners, and provides **advanced configuration** options to manage payment rules for specific URI patterns. Extensive **documentation updates** clarify payment header signing, introduce a new `crawler-error` header, detail error codes, and explain always-free routes and in-band pricing behavior. These **new capabilities** provide both AI owners and site owners with more granular control and clearer guidance for interacting with the Pay Per Crawl ecosystem.

This commit delivers a **hotfix** to the **AI Crawl Control** documentation, specifically addressing inaccuracies in the `pay-per-crawl` feature guide. It **corrects example code** within `src/content/docs/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages.mdx`, updating the recommended HTTP status code from `200` to `402` and the header name from `crawler-charged` to `crawler-price`. This **documentation improvement** ensures developers receive precise guidance for implementing the pay-per-crawl mechanism, preventing potential integration errors due to outdated or incorrect examples.

This commit provides **documentation** and **changelog updates** for new features within the **AI Crawl Control** product. It details the introduction of **crawler drilldowns**, an **extended actions menu**, and **status code analytics** for monitoring AI traffic. This **maintenance** work ensures users have comprehensive guides on how to leverage these new capabilities, enhancing their ability to analyze and manage AI crawler interactions.

This commit delivers **essential documentation** and a **changelog entry** for the newly introduced 'Robots.txt' tab within the **AI Crawl Control** feature. It provides **new documentation** for the `Track robots.txt` functionality, detailing how users can monitor availability, filter data, and track violations of their `robots.txt` files. Furthermore, it **updates existing documentation** for AI traffic analysis with new sections on referrer data and crawler requests, and adds a feature card for `Track robots.txt` to the main `AI Crawl Control` overview. This **documentation update** significantly enhances user understanding and discoverability of the latest AI crawler management capabilities.

This commit **adds a new changelog entry** for the **AI Crawl Control** feature, specifically documenting the introduction of **enhanced metrics, drilldowns, and data export capabilities**. This is a **documentation update** to inform users about significant new functionalities within the AI Crawl Control system. The entry, located at `src/content/changelog/ai-crawl-control/2025-10-14-enhanced-metrics-drilldowns.mdx`, provides details on these improvements, including updated language and getting started steps.

This commit delivers significant **documentation updates** and **refactoring** for the **AI Crawl Control** feature, ensuring the content accurately reflects recent product changes. It updates existing documentation to align with dashboard UI changes, such as renaming the 'AI Crawlers' tab to 'Crawlers' and introducing new metrics like 'Unsuccessful requests'. Crucially, this work introduces comprehensive new documentation for the **Pay Per Crawl site owner onboarding flow**, detailing steps for enabling the feature, managing payouts, monitoring activity, and setting pricing. Additionally, it includes a new dedicated FAQ for Pay Per Crawl, fixes missing internal links, removes duplicate content, and establishes redirects to maintain consistent navigation for users.