feat: add agentcore-observability-setup skill

cloudformation/devops-agent-skill-policies.yaml

@@ -27,6 +27,7 @@ Metadata:
           - EnableMskOperations
           - EnableServiceQuotaCheck
           - EnableDmsOperationReview
+          - EnableAgentCoreObservabilitySetup
       - Label:
           default: Optional Resource Scoping
         Parameters:
@@ -111,6 +112,12 @@ Parameters:
     AllowedValues: ['true', 'false']
     Default: 'true'
 
+  EnableAgentCoreObservabilitySetup:
+    Type: String
+    Description: AgentCore Observability Setup skill (adds read-only bedrock-agentcore, X-Ray, log-delivery, and Lambda/ECS/EKS host-config permissions).
+    AllowedValues: ['true', 'false']
+    Default: 'true'
+
 Conditions:
   CreateNewRole: !Equals [!Ref ExistingRoleName, '']
   SkillAwsHealthEvents: !Equals [!Ref EnableAwsHealthEvents, 'true']
@@ -120,6 +127,7 @@ Conditions:
   SkillMskOperations: !Equals [!Ref EnableMskOperations, 'true']
   SkillServiceQuotaCheck: !Equals [!Ref EnableServiceQuotaCheck, 'true']
   SkillDmsOperationReview: !Equals [!Ref EnableDmsOperationReview, 'true']
+  SkillAgentCoreObservabilitySetup: !Equals [!Ref EnableAgentCoreObservabilitySetup, 'true']
   HasRegionRestriction: !Not [!Equals [!Join ['', !Ref AllowedRegions], '']]
 
 Resources:
@@ -302,6 +310,44 @@ Resources:
               - dms:TestConnection
             Resource: '*'
 
+  # agentcore-observability-setup: adds read-only AgentCore control-plane, X-Ray Transaction Search,
+  # log-delivery/resource-policy describes, and Lambda/ECS/EKS host-config reads (Tier 2 + Tier 3).
+  # Tier 1 (CloudWatch Logs/Metrics telemetry-arrival reads) is covered by AIDevOpsAgentAccessPolicy.
+  PolicyAgentCoreObservabilitySetup:
+    Type: AWS::IAM::Policy
+    Condition: SkillAgentCoreObservabilitySetup
+    Properties:
+      PolicyName: DevOpsAgentSkill-AgentCoreObservabilitySetup
+      Roles:
+        - !If [CreateNewRole, !Ref DevOpsAgentRole, !Ref ExistingRoleName]
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Sid: AgentCoreRuntimeConfigRead
+            Effect: Allow
+            Action:
+              - bedrock-agentcore:GetAgentRuntime
+              - bedrock-agentcore:ListAgentRuntimes
+            Resource: '*'
+          - Sid: XRayTransactionSearchAndDeliveries
+            Effect: Allow
+            Action:
+              - xray:GetTraceSegmentDestination
+              - logs:DescribeDeliveries
+              - logs:DescribeDeliverySources
+              - logs:DescribeDeliveryDestinations
+              - logs:DescribeResourcePolicies
+            Resource: '*'
+          - Sid: NonRuntimeHostConfigRead
+            Effect: Allow
+            Action:
+              - lambda:GetFunctionConfiguration
+              - ecs:DescribeTaskDefinition
+              - ecs:DescribeServices
+              - ecs:ListTasks
+              - eks:DescribeCluster
+            Resource: '*'
+
   # Optional: restrict agent to specific regions
   PolicyRegionalRestriction:
     Type: AWS::IAM::Policy
@@ -349,6 +395,7 @@ Outputs:
         - msk-operations: ${EnableMskOperations} (kafka:GetBootstrapBrokers)
         - service-quota-check: ${EnableServiceQuotaCheck} (servicequotas:*, cloudwatch:GetMetricData/GetMetricStatistics)
         - database-migration-service-expertise: ${EnableDmsOperationReview} (dms:TestConnection)
+        - agentcore-observability-setup: ${EnableAgentCoreObservabilitySetup} (bedrock-agentcore:Get/ListAgentRuntime, xray:GetTraceSegmentDestination, logs:DescribeDeliveries/DeliverySources/DeliveryDestinations/ResourcePolicies, lambda:GetFunctionConfiguration, ecs:DescribeTaskDefinition/DescribeServices/ListTasks, eks:DescribeCluster)
       Skills covered by AIDevOpsAgentAccessPolicy (no extra policy needed):
         - eks-operation-review, enrich-with-aws-security-agent, crm-production-investigation-guidelines
       No IAM required:

llms.txt

@@ -27,6 +27,7 @@ Skills can be used with these AWS DevOps Agent types:
 - [S3 Resiliency Review Skill](skills/storage-s3-resiliency-expertise/SKILL.md): Reviews one or many S3 buckets across nine resiliency, security, and data-protection dimensions using read-only control-plane calls, producing a rated report with prioritized findings and remediation guidance
 - [VPC DNS Investigation Skill](skills/aws-vpc-dns-investigation/SKILL.md): Diagnoses VPC DNS resolution failures and validates DNS control-plane changes before they are applied, driving the aws-vpc-dns-diagnostics MCP server to observe live resolution from inside the affected subnet and to simulate a proposed change
 - [Bedrock Adoption Readiness Skill](skills/bedrock-adoption-readiness/SKILL.md): Assesses an AWS account's readiness to run Amazon Bedrock at production scale across IAM governance, data retention (ZDR), quota and capacity headroom, and operational observability, covering both the standard Bedrock and bedrock-mantle (OpenAI-compatible) surfaces with multi-region discovery
+- [AgentCore Observability Setup Skill](skills/agentcore-observability-setup/SKILL.md): Validates and bootstraps Amazon Bedrock AgentCore observability across runtime agents, Memory and Gateway resources, built-in tools, and agents hosted outside the runtime, verifying telemetry wiring via read-only CloudWatch, X-Ray, and AgentCore APIs and prescribing exact remediation for gaps it cannot directly read
 
 ## Key Concepts
 

skills/agentcore-observability-setup/.skilleval.yaml

@@ -0,0 +1,3 @@
+audit:
+  ignore:
+    - STR-016    # README alongside SKILL.md is intentional

skills/agentcore-observability-setup/CHANGELOG.md

@@ -0,0 +1,12 @@
+# Changelog
+
+## 1.0.0
+
+- Initial version.
+- Validates and bootstraps Amazon Bedrock AgentCore observability using a verify-where-reachable /
+  prescribe-everywhere-else model.
+- Covers Runtime agents, Memory and Gateway resources, built-in tools, and non-runtime hosts
+  (Lambda, ECS, EKS, on-prem, multi-cloud).
+- Three read-only IAM tiers with graceful degradation and scoped inline policy JSON.
+- Check catalog and per-host remediation references (Transaction Search, ADOT SDK, unified span
+  destination, X-Ray log-group resource policy, Memory/Gateway log & trace delivery).

skills/agentcore-observability-setup/README.md

@@ -0,0 +1,114 @@
+# AgentCore Observability Setup & Validation — AWS DevOps Agent Skill
+
+A readiness-checklist skill for [AWS DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/about-aws-devops-agent.html)
+that validates and bootstraps observability for Amazon Bedrock AgentCore workloads, aligned with the
+[AgentCore observability documentation](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html).
+
+> ⚠️ This skill is sample code, not intended for production use without additional review and
+> testing. Users should validate in a non-production environment first.
+
+## What It Does
+
+Given an AgentCore deployment, the skill instructs the agent to report **what is configured vs. what
+should be** for observability, and to generate the exact remediation for each gap. It operates on a
+single principle: **verify where reachable; prescribe everywhere else.**
+
+- **VERIFY** — confirm via read-only APIs that log groups exist and receive data, spans flow,
+  session metrics emit, CloudWatch Transaction Search is enabled, runtime tracing is on,
+  Memory/Gateway log & trace deliveries exist, and the X-Ray log-group resource policy is present.
+- **PRESCRIBE** — for code-level OTEL instrumentation (which cannot be read directly), validate the
+  *effect* (are spans arriving?) and emit the exact steps. The skill does **not** claim to read
+  source code.
+
+Covers AgentCore **Runtime** agents (primary), **Memory** and **Gateway** resources, **built-in
+tools**, and agents hosted **outside the runtime** (Lambda, ECS, EKS, on-prem, multi-cloud).
+
+It is **read-only**: it validates and generates config/commands. The customer decides whether to
+apply them. It never mutates IAM, resources, or configuration.
+
+Output: a shareable report artifact `agentcore-observability-review-<target>-<YYYY-MM-DD>.md`.
+
+## Agent Types
+
+- **On-demand (Chat)** — "is AgentCore observability configured correctly?", "why can't I see traces
+  for my agent?", "set up OTEL for my agent".
+- **Evaluation** — proactive observability-readiness recommendations.
+
+Select **Generic** to make it available to all agent types.
+
+## Prerequisites
+
+### 1. An AWS DevOps Agent Space with the target AWS account configured as a cloud source.
+
+### 2. IAM — three read-only tiers (customer-applied)
+
+The skill degrades gracefully by tier and reports the exact policy to attach to unlock more checks.
+Tier 1 is generally covered by the standard DevOps Agent managed access policy
+([AWS-managed policies for DevOps Agent](https://docs.aws.amazon.com/devopsagent/latest/userguide/security-iam-awsmanpol.html)).
+Attach the Tier 2/3 permissions as a **scoped inline policy on the DevOps Agent role** in your
+account. The shared managed policy is cross-tenant and is not modified per-skill.
+
+| Tier | Unlocks | Key permissions |
+|---|---|---|
+| **1 — Standard** | telemetry-arrival verification (any host) | `logs:DescribeLogGroups`, `logs:FilterLogEvents`, `logs:GetLogEvents`, `logs:StartQuery`, `cloudwatch:GetMetricData`, `cloudwatch:ListMetrics`, `cloudwatch:DescribeAlarms` |
+| **2 — Runtime config** | Transaction Search state, runtime tracing/env, Memory/Gateway delivery, X-Ray resource policy | `bedrock-agentcore:GetAgentRuntime`, `xray:GetTraceSegmentDestination`, `logs:DescribeDeliveries`, `logs:DescribeResourcePolicies` |
+| **3 — Non-runtime host** | Lambda / ECS / EKS host config | `lambda:GetFunctionConfiguration`, `ecs:DescribeTaskDefinition`, `eks:DescribeCluster` |
+
+Full policy JSON: [`references/iam-tiers.md`](references/iam-tiers.md). All permissions are read-only.
+
+### 3. Capability providers connected to the Agent Space
+
+CloudWatch Logs + Metrics (Tier 1, required baseline); AWS X-Ray and AgentCore control plane
+(Tier 2); Lambda / ECS / EKS (Tier 3). Source-repository file read is **not** required — code-level
+checks are prescriptive.
+
+## Usage
+
+In the DevOps Agent Chat, describe the problem in natural language (don't name the skill):
+
+- *"Validate that AgentCore observability is correctly configured for this account."*
+- *"Why can't I see any traces for my Bedrock agent?"*
+- *"My agent runtime has no spans in CloudWatch — what's wrong?"*
+- *"Set up observability for my agent running on Lambda."*
+- *"Run an AgentCore observability readiness review."*
+
+The agent will detect the permission tier, determine the host and surfaces (prompting where it can't
+auto-detect), run the checks the tier allows, and produce the report artifact plus any policy JSON
+needed to unlock more checks.
+
+## Skill Contents
+
+```
+agentcore-observability-setup/
+├── SKILL.md                                  # main instructions (frontmatter + decision tree)
+├── README.md                                 # this file
+├── CHANGELOG.md                              # version history
+├── references/
+│   ├── checks-catalog.md                     # full per-check catalog (API, logic, severity)
+│   ├── iam-tiers.md                          # three read-only tiers + scoped inline policy JSON
+│   ├── remediation-runtime.md                # Transaction Search, ADOT, tracing, X-Ray policy
+│   ├── remediation-memory-gateway.md         # log delivery + tracing (console + SDK)
+│   └── remediation-non-runtime.md            # Lambda / ECS / EKS / on-prem OTEL env
+└── evals/                                    # evaluation data (not included in upload zip)
+    ├── evals.json
+    └── eval_queries.json
+```
+
+## Severity Definitions
+
+| Severity | Definition |
+|---|---|
+| CRITICAL | Observability is broken — telemetry cannot flow (e.g. Transaction Search disabled). |
+| HIGH | Significant gap — a required piece is missing (log group, delivery, resource policy, host env). |
+| MEDIUM | Notable gap — metrics/traces partially configured or best-practice not met. |
+| LOW | Minor hardening or optimization. |
+| INFO | Observation, no action required. |
+
+## Limitations
+
+- **Code-level instrumentation** (ADOT distro, `opentelemetry-instrument`, framework tracing) is
+  prescriptive — validated by effect (spans arriving), not by reading source.
+- **EKS pod-level env** is governed by Kubernetes RBAC and is not verifiable via IAM; the skill
+  verifies telemetry arrival and prescribes the pod/ConfigMap configuration.
+- **On-prem / multi-cloud** agents are outside the DevOps Agent's reach — prescriptive only.
+- **Read-only** — the skill validates and generates config/commands; the customer applies changes.