Skip to content

feat: add agentcore-observability-setup skill - #65

Open
vipulg1993 wants to merge 1 commit into
aws:mainfrom
vipulg1993:feature/agentcore-observability-setup
Open

feat: add agentcore-observability-setup skill#65
vipulg1993 wants to merge 1 commit into
aws:mainfrom
vipulg1993:feature/agentcore-observability-setup

Conversation

@vipulg1993

Copy link
Copy Markdown

Summary

Adds the agentcore-observability-setup skill: validates and bootstraps Amazon Bedrock AgentCore observability so customers can trace agent reasoning, detect silent failures, and measure performance before an outage. It operates on a single principle — verify where reachable, prescribe everywhere else:

  • VERIFY (read-only APIs): CloudWatch Transaction Search state, runtime/agent log groups and data arrival, spans flowing, session metrics, Memory/Gateway log & trace deliveries, X-Ray log-group resource policy, and Lambda/ECS host config.
  • PRESCRIBE (cannot be read directly): code-level OTEL instrumentation (ADOT distro, opentelemetry-instrument, framework tracing) — validated by effect (are spans arriving?) and emitted as exact steps.

Covers AgentCore Runtime agents (primary), Memory and Gateway resources, built-in tools, and agents hosted outside the runtime (Lambda, ECS, EKS, on-prem, multi-cloud). Three read-only IAM tiers with graceful degradation; the skill reports the exact scoped policy needed to unlock further checks and never mutates anything.

Testing

  • Manual testing in DevOps Agent (Isengard): skill activated on natural-language prompts (not named), reached Tier 2, and produced correct findings on both a healthy account and an induced-fault scenario (Transaction Search disabled), where it identified the disabled Transaction Search as the CRITICAL root cause and prescribed remediation.
  • Agent Skill Eval (report): overall Grade A, "passed": true. Audit 100/A (0 critical/warning), Trigger 1.00 (6/6, 100% precision both ways), Functional passed (quality gain with lower token cost).

IAM / CloudFormation

Tier 1 (CloudWatch Logs/Metrics telemetry-arrival reads) is covered by AIDevOpsAgentAccessPolicy. Tier 2/3 add read-only bedrock-agentcore (Get/List runtime), xray:GetTraceSegmentDestination, logs:DescribeDeliveries/DeliverySources/DeliveryDestinations/ResourcePolicies, and lambda:GetFunctionConfiguration / ecs:DescribeTaskDefinition,DescribeServices,ListTasks / eks:DescribeCluster. Added as a gated inline policy (EnableAgentCoreObservabilitySetup, default true) to cloudformation/devops-agent-skill-policies.yaml, with the SkillPolicySummary output updated. All permissions are read-only.

Taskei

V2323594559 — accepted by the Project Nebula team (reviewer: pbraz).

Checklist

  • SKILL.md has version and author in metadata
  • name in frontmatter matches directory name
  • description clearly states what/when to activate
  • README.md includes non-production disclaimer
  • CHANGELOG.md present
  • Tested in DevOps Agent (skill activates consistently)
  • Agent Skill Eval "passed": true (Grade A)
  • Registered in llms.txt
  • .skilleval.yaml present (canonical STR-016 ignore)
  • No internal Amazon tool names in submitted content
  • No customer data, account IDs, or credentials
  • Working against latest main

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

Add a skill that validates and bootstraps Amazon Bedrock AgentCore
observability using a verify-where-reachable / prescribe-everywhere-else
model. Covers Runtime agents, Memory and Gateway resources, built-in
tools, and non-runtime hosts (Lambda, ECS, EKS, on-prem, multi-cloud)
with three read-only IAM tiers and graceful degradation.

Register the skill in llms.txt and add a gated read-only inline policy
(EnableAgentCoreObservabilitySetup) to the CloudFormation skill policies
for the Tier 2/3 permissions not covered by AIDevOpsAgentAccessPolicy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant