AWS and Arize AI Enhance Amazon Bedrock Agents with Observability

Amazon Web Services (AWS) and Arize AI have unveiled a new integration designed to enhance the observability of Amazon Bedrock Agents, addressing a critical challenge in AI development. This collaboration provides developers with comprehensive tools to trace, evaluate, and optimize AI agent performance, ensuring reliability and efficiency in production environments.

Key Features of the Integration

The integration leverages Arize Phoenix, an open-source system for tracing and evaluation, to offer:

• Comprehensive Traceability: Visibility into every step of an agent’s execution, from user queries to API calls and tool invocations.
• Systematic Evaluation Framework: Consistent methodologies to measure agent performance, including function calling accuracy and path convergence.
• Data-Driven Optimization: Structured experiments to compare agent configurations and identify optimal settings.

How It Works

The solution uses OpenInference, a set of instrumentations for ML frameworks, to automatically capture traces of Amazon Bedrock Agent interactions. These traces are then visualized in the Phoenix dashboard, enabling developers to:

• Monitor latency, token usage, and runtime exceptions.
• Inspect retrieved documents, embeddings, and LLM parameters.
• Debug inefficient paths or unexpected behaviors in agent decision-making.

Evaluation Capabilities

Phoenix includes built-in LLM-as-a-Judge templates to evaluate agent performance, such as:

• Agent Function Calling: Measures how well the agent selects and executes tools.
• Agent Path Convergence: Evaluates whether the agent takes optimal paths to solutions.
• Agent Planning and Reflection: Assesses the agent’s reasoning and adaptability.

Developers can log evaluation results directly to Phoenix, gaining actionable insights to refine agent behavior.

Getting Started

To implement this integration, users need:

1. An AWS account with access to Amazon Bedrock.
2. An Amazon Bedrock Agent (created manually or via AWS CDK).
3. An Arize account for Phoenix API keys.

The GitHub repo provides sample code and Jupyter notebooks to streamline setup.

Conclusion

This integration marks a significant step forward in AI agent observability, empowering developers to build more reliable and performant generative AI applications. For more details, explore the Phoenix documentation or read the Arize AI blog.

Authored by AWS and Arize AI specialists, including Ishan Singh, John Gilhuly, and Richa Gupta.