Tao Chen

This commit introduces **support for the new OpenAI text-to-image model** (`gpt-1-image`) within the **Python OpenAI connector**, addressing the deprecation of older DALL-E models. It **modifies the `generate_image` method** in `open_ai_text_to_image_base.py` to return base64 encoded image data when a direct URL is not available from the service. This **new capability** ensures compatibility with the latest OpenAI API, impacting how generated images are consumed by applications. The `image_generation.py` sample has been updated to reflect this change, demonstrating how to handle base64 image responses. Additionally, it includes a minor **refactoring** of `open_ai_text_to_image_execution_settings.py` for parameter types and adds comprehensive unit tests.

This commit significantly **improves the robustness and security of file handling** within the Python Semantic Kernel. It introduces **filename sanitization** for responses received from the **Bedrock agent integration**, preventing potential directory traversal vulnerabilities and ensuring safe file operations. Additionally, the `BinaryContent.write_to_file` method now includes an `overwrite` parameter, defaulting to `False`, to provide a **safer default behavior** and prevent accidental data loss. This **security enhancement** includes necessary unit tests for the new sanitization logic and file write behavior, along with updates to Bedrock agent samples to reflect these improvements.

This commit **migrates the Python Google AI connector** within `semantic_kernel.connectors.ai.google` to utilize the **new Google GenAI SDK**, significantly updating the underlying implementation for chat completion, text completion, and text embedding services. As part of this **refactoring and enhancement**, the existing **Vertex AI connector is deprecated**, with users now directed to configure the unified Google AI connector for Vertex AI projects, and `AzureTextCompletion` is also marked for deprecation. The update also introduces the capability to provide a **custom client** to the Google AI connector, improving flexibility. This change impacts various samples, documentation, and tests, streamlining the Google AI integration and requiring users to adapt to the new unified approach for Google and Vertex AI services.

This commit **refactors** the **OpenTelemetry integration** for **Generative AI operations** across both the **.NET** and **Python** SDKs. It updates the standard operation names used for tracing chat and text completion activities, specifically changing `chat.completion` to `chat` and `text.completion` to `text_completion`. This **maintenance** effort ensures consistency and aligns with updated OpenTelemetry naming conventions for GenAI spans, directly impacting how telemetry data is reported and consumed by monitoring systems.

This commit introduces a **new capability** to enhance **diagnostic tracing** within the **.NET Semantic Kernel**. It specifically adds the `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` attributes to the `execute_tool` activity span, providing more detailed observability for **kernel function invocations** and **tool execution**. These attributes are only included when sensitive data logging is explicitly enabled, ensuring data privacy by default. The changes primarily affect the **`KernelFunction` abstraction** in `SemanticKernel.Abstractions` and related **diagnostic utilities**, improving the depth of information available for monitoring and debugging tool calls.

This commit delivers a **bug fix** for **.Net GenAI telemetry** serialization, resolving an issue where certain GenAI content was not correctly formatted as JSON. The change refactors methods such as `LogChatCompletion`, `ToGenAIConventionsFormat`, and `ToGenAIConventionsChoiceFormat` within `ModelDiagnostics.cs` to ensure they directly produce properly structured JSON objects. This guarantees that **GenAI content is accurately serialized and logged**, enabling correct consumption and analysis by external monitoring tools.

This commit **enhances telemetry for agent invocations** in both the **.NET and Python Semantic Kernel libraries** by aligning with updated GenAI Semantic Conventions. It **refactors existing OpenTelemetry spans** by renaming `gen_ai.agent.invocation_input` to `gen_ai.input.messages` and `gen_ai.agent.invocation_output` to `gen_ai.output.messages`. Crucially, it **adds a new `gen_ai.tool.definitions` attribute** to the `agent_invoke` span, providing richer diagnostic information about the tools available to the agent. This **feature enhancement** impacts the **diagnostics and observability** of agent interactions across various agent implementations, ensuring more detailed and standardized tracing data is captured.

This commit introduces a **bug fix** and an **enhancement** to the **Python Azure AI Inference connector**. It resolves an issue where `extra_parameters` were duplicated in chat completion requests, causing unknown parameter errors, by ensuring the `extra_parameters` key is correctly handled in `AzureAIInferenceChatPromptExecutionSettings`. Concurrently, this update **enhances the Azure AI Inference services** by allowing users to explicitly set the `api_version` for both chat completion and text embedding, providing greater control over API interactions. This improves the reliability and flexibility of the **Azure AI Inference integration**, preventing API errors and enabling version-specific API calls. New tests and a sample demonstrating reasoning with Azure AI Inference are also included.

This commit introduces **enhanced OpenTelemetry tracing** for **.NET agents** by adding `gen_ai.agent.invocation_input` and `gen_ai.agent.invocation_output` attributes to the `invoke_agent` spans. This **new capability** provides richer observability into agent interactions, capturing the full input and output of agent invocations directly within telemetry traces. The change affects various agent implementations (e.g., `AzureAIAgent`, `BedrockAgent`, `ChatCompletionAgent`) and the internal diagnostics utilities, improving **debugging and monitoring** of agent workflows. This is a **feature enhancement** that aligns with a proposed OpenTelemetry semantic convention for agent tracing, ensuring future compatibility and standardized observability. A new example in `Program.cs` demonstrates this improved telemetry for `ChatCompletionAgent` runs.

This commit **enhances OpenTelemetry tracing** within the **Python Semantic Kernel** by adding `gen_ai.tool.call.arguments` and `gen_ai.tool.call.result` attributes to the `execute_tool` span. This **new capability** improves **observability** of tool invocations by capturing detailed input arguments and results, aligning with proposed OpenTelemetry semantic conventions. The `KernelFunction`'s `invoke` and `invoke_stream` methods are updated to record these attributes, supported by a new `dumps` method in `KernelArguments` for serialization and a mechanism to control sensitive event logging. This **feature addition** provides richer diagnostic data for understanding tool execution flows and ensures compliance with evolving telemetry standards.

This commit introduces a **new capability** to the **Python Semantic Kernel's Bedrock AI connectors**, allowing users to explicitly specify the `model_provider` (e.g., `amazon`, `anthropic`) during service initialization. This **feature enhancement** addresses issue #12833 by enabling the use of **application inference profiles** within **Bedrock chat completion**, **text completion**, and **text embedding** services. The change impacts the `bedrock_settings.py` and various `bedrock` service classes, providing greater flexibility and control over Bedrock model interactions. This update includes comprehensive unit tests and documentation to support the new configuration option.

This commit introduces a **feature enhancement** to the **OpenTelemetry tracing** for **agents** within Semantic Kernel, adding `gen_ai.agent.invocation_input` and `gen_ai.agent.invocation_output` attributes to the `invoke_agent` spans. The `semantic_kernel.utils.telemetry.agent_diagnostics` module is **refactored** to include new decorators like `trace_agent_streaming_invocation`, which conditionally capture these invocation details and errors. Various **agent implementations** (e.g., Azure AI, OpenAI) are updated to utilize these enhanced tracing capabilities, providing richer diagnostic data for agent interactions. This work proactively aligns with proposed OpenTelemetry semantic conventions, significantly improving the **observability** and debugging experience for agent-based applications.

This commit introduces a **feature enhancement** to the **.NET Magentic orchestration**, specifically within `MagenticManagerActor.cs`. Previously, when maximum invocation or reset limits were reached, a static message was returned; now, the system will **return the last agent message** as a partial result. This provides more informative feedback to users, improving the debugging experience and robustness of agent interactions. Additionally, the **Python Magentic manager** (`magentic.py`) has a minor **maintenance update** to change the logging level from debug to error when similar limits are hit.

This commit introduces a **new sample** demonstrating **multi-agent orchestration** with mixed agent types in a handoff scenario, enhancing the `python/samples` module. It also includes two critical **bug fixes**: one in the **`OpenAIResponsesAgent`** to ensure tools are correctly recognized from the kernel instance via the `_get_tools` method, and another in the **`AgentActorBase`** to properly handle `ChatMessageContent` in intermediate message callbacks through `_handle_intermediate_message`. A new unit test was added to validate the `OpenAIResponsesAgent` fix, improving the reliability of agent tool integration and communication within the `semantic_kernel`'s agent framework. This work provides clearer guidance for complex agent interactions and enhances the robustness of agent communication.

This commit **improves exception handling** within the **Python orchestration framework** by ensuring that exceptions originating from agent actors are properly surfaced to the caller, addressing a critical issue where errors were previously swallowed. It introduces a new, internal `exception_callback` mechanism and an `exception_handler` decorator in `agent_actor_base.py` to facilitate this. This **new capability** is integrated across various **orchestration types**, including `Concurrent`, `GroupChat`, `Handoff`, `Magentic`, and `Sequential` orchestrations, and enhances `OrchestrationResult` with a `background_task` field. The change significantly **enhances the reliability and debuggability** of agent orchestrations, allowing developers to catch and react to errors more effectively. New unit tests have been added to validate the correct propagation of exceptions across these diverse orchestration types.

This commit introduces a **message cache** within **Python agent orchestrations** to temporarily store incoming messages before an agent is invoked. This **refactoring** addresses compatibility issues with agents, such as the **Copilot Studio agent**, that do not support direct message additions to their threads. Specifically, files like `agent_actor_base.py`, `group_chat.py`, `handoffs.py`, and `magentic.py` are updated to utilize this cache, streamlining message handling and removing conditional logic related to agent threads. The change ensures that agents receive all accumulated messages at once, improving the robustness and flexibility of the **agent orchestration** system.

This commit introduces a **new capability** to the **`semantic_kernel.agents`** module, enabling **agent response callbacks** to be triggered by **intermediate messages** during multi-agent orchestrations. Previously, callbacks only fired on final responses, but now they provide full context of an agent's internal processing, including function calls and results. This enhancement, implemented across `agent_actor_base.py` and `handoffs.py`, significantly improves observability and control over agent interactions. Additionally, a **bug fix** ensures the agent's name is correctly assigned to these intermediate messages in `chat_completion_agent.py`. Updated **samples** and comprehensive **unit tests** validate this expanded functionality, allowing developers to build more transparent and sophisticated multi-agent systems.

This commit **corrects and refines the Python samples** for the **chat completion agent's history truncation reducer**, addressing issues identified in a recent discussion. Specifically, it fixes a typo in the `history_truncatation_reducer` variable name and removes an unnecessary `service` parameter from the `ChatHistoryTruncationReducer` constructor within the example code. This **maintenance** and **bug fix** ensures the provided samples in `python/samples/concepts/agents/chat_completion_agent/` are accurate and easier for developers to understand and implement. The changes improve the clarity and correctness of the example usage for the **chat completion agent's history management** within the `semantic-kernel` library.

This commit **refactors** the **Python agent orchestration** system by **removing the persona adaptation messages** from the `group_chat`, `handoffs`, and `magentic` components. These messages, previously used to steer agents, are now deemed unnecessary with newer models, introducing instability and token cost without sufficient benefit. This **maintenance** task **simplifies agent behavior** in multi-agent scenarios, making the system more robust and easier to debug by eliminating hardcoded, unoverrideable prompts. Unit tests for these orchestration modules were updated to reflect the change, and some samples were also adjusted to use `AzureChatCompletion`.

This commit **enhances** the **Python Bedrock AI connectors** by **removing model capability checks** for streaming and embedding generation. Previously, these checks in `bedrock_text_completion.py` and `bedrock_text_embedding.py` relied on a specific `model_id`, which limited flexibility when identifying models via `inference_profile` or `model ARN`. This **refinement** allows users to specify Bedrock models using a wider range of identifiers, making the **text completion** and **text embedding** services more adaptable. The change improves the overall robustness and usability of the Bedrock integration by enabling more flexible model identification.