Introducing Amazon CloudWatch Omni: AI-powered observability for generative AI and agentic workloads

Summary

• Amazon CloudWatch Omni is a unified, app-centric, AI-powered observability solution for application and AI workloads delivered off-console.
• The solution provides a purpose-built environment for designing, evaluating, and operating AI agents across any model provider, framework, or runtime.
• Observability is delivered directly within IDEs like VS Code and Kiro, as well as through a standalone web experience separate from the AWS Management Console.
• CloudWatch Omni captures every trace and includes built-in evaluators for correctness, coherence, retrieval quality, and tool selection.
• Developers can compare prompt versions side by side, build test datasets from production traffic, run experiments, and detect regressions automatically.
• The platform offers two complementary surfaces: a native IDE extension for developers and a standalone web experience for operators.
• Both surfaces share the same trace data, ensuring that a trace debugged by a developer is the same trace investigated by an operator.
• The Cloud Login feature connects local IDE environments to AWS accounts for persistent storage, team sharing, and production dashboard access, though this connection is optional.
• Users can start via the IDE extension or directly through the cloud experience without installing any IDE extension.
• The setup process includes adding OpenTelemetry instrumentation, with guidance provided through an interactive chat or AI code assistants like Kiro, Claude Code, and Codex.
• The Trace Explorer displays a structured, hierarchical timeline of LLM calls, tool invocations, and reasoning steps with drill-down capabilities for inputs, outputs, token usage, and latency.
• Compare mode allows side-by-side analysis of two traces to evaluate the impact of different prompts or configurations on agent behavior.
• The Ask Assistant feature uses an AI agent to analyze traces and surface patterns or anomalies, such as explaining why an agent called a specific tool multiple times.
• The platform includes 17 built-in evaluators for metrics like coherence, helpfulness, faithfulness, and routing correctness to measure user experience quality.
• The Playground enables real-time testing of different system prompts, model configurations, and agent variants to assess output quality before committing changes.
• The Experiments view allows teams to run the same dataset against multiple agent variants to compare evaluation scores, latency, and token usage.
• Prompt Management features allow versioning and tracking of prompt configurations over time to facilitate rollbacks when new versions underperform.
• The Session Explorer reviews full conversation histories for multi-turn interactions, while the Agent Topology view visualizes system architecture including sub-agents, tools, and interconnections.
• The Experiment function runs agents against curated golden datasets to automatically score results and create benchmarks for regression testing.
• Instrumentation is supported via auto-instrumentation with Kiro or manual instrumentation using code snippets for Python and TypeScript.
• CloudWatch Omni supports frameworks including LangChain, LangGraph, CrewAI, OpenAI SDK, Strands, Vercel AI SDK, and Amazon Bedrock AgentCore.
• Instrumentation utilizes open standards such as OpenInference and ADOT for agents running on Lambda, ECS, EKS, or other clouds.
• Evaluation integrations include third-party tools like AutoEval and DeepEval alongside built-in datasets and batch experiments.
• Amazon CloudWatch Omni is now generally available with a free IDE extension that does not require an AWS account to start.
• AWS credentials are only needed for Amazon Bedrock models, while API keys are required for other providers like OpenAI or Anthropic.