Store credentials in Secrets Manager consistently + configurable KMS key

aurora-postgres-db/README.md

@@ -1,13 +1,13 @@
 # aurora-postgres-db
 
-A nullplatform dependency service that provisions and manages a **PostgreSQL database** within an existing Aurora cluster managed by [`aurora-postgres-server`](../aurora-postgres-server). It handles database creation, app-level user management, and per-link fine-grained access control — without creating any AWS infrastructure itself.
+A nullplatform dependency service that provisions and manages a **PostgreSQL database** within an existing Aurora cluster managed by [`aurora-postgres-server`](../aurora-postgres-server). It handles database creation, app-level user management, and per-link fine-grained access control. Unlike `aurora-postgres-server`, it does not provision any Aurora/EC2 infrastructure — the one AWS resource it does create is the Secrets Manager secret holding the app-level PostgreSQL credentials.
 
 ## What It Does
 
 - Auto-discovers a compatible `aurora-postgres-server` in the same nullplatform namespace using dimension matching **and** an internal `engine_family = "aurora-postgresql"` attribute (disambiguates from a classic RDS `rds-postgres-server` that might share the same dimensions)
 - Creates a dedicated PostgreSQL database and application-level user within that cluster
 - Manages per-link permissions: each link to an application applies scoped grants (`read`, `write`, or `read-write`) to the single, shared service-level PostgreSQL user — links do not get their own user, only their own grant set
-- Stores connection credentials in nullplatform service and link attributes for injection into applications
+- Stores connection credentials in nullplatform service and link attributes for injection into applications, and mirrors the app-level credentials into a Secrets Manager secret (same convention as the `aurora-postgres-server` master secret)
 
 ## Architecture
 
@@ -24,7 +24,7 @@ nullplatform Application
              postgresql_grant.*  (on the existing service-level user — no new role is created)
 ```
 
-Unlike `aurora-postgres-server`, this service creates no AWS resources. It only manages PostgreSQL-level objects on the shared Aurora cluster: one database and one role at service level, plus per-link grants on that same role — unlike `aurora-postgres-server`, which creates a dedicated role per link.
+Unlike `aurora-postgres-server`, this service creates no Aurora/EC2 infrastructure — it only manages PostgreSQL-level objects on the shared Aurora cluster: one database and one role at service level, plus per-link grants on that same role, plus one Secrets Manager secret for the app-level credentials it generates.
 
 ## Nullplatform Integration
 
@@ -42,7 +42,8 @@ Unlike `aurora-postgres-server`, this service creates no AWS resources. It only
 | `username` | exported | Service-level PostgreSQL user |
 | `password` | hidden | Service-level PostgreSQL password |
 | `database_name` | exported | PostgreSQL database name |
-| `master_secret_arn` | internal | Secrets Manager ARN (used for link operations) |
+| `master_secret_arn` | internal | Secrets Manager ARN for the `aurora-postgres-server` master credentials (used for link operations) |
+| `app_secret_arn` | internal | Secrets Manager ARN for this service's own app-level credentials |
 
 ### Link Attributes (written per link)
 
@@ -51,6 +52,7 @@ Unlike `aurora-postgres-server`, this service creates no AWS resources. It only
 | `username` | The service-level PostgreSQL user, mirrored to the link (same value for every link on this service) |
 | `password` | The service-level PostgreSQL password, mirrored to the link |
 | `database_name` | Database name (same as service-level database) |
+| `app_secret_arn` | Secrets Manager ARN for the service-level app credentials (mirrored from the service attribute) |
 
 ## Link Parameters
 
@@ -70,9 +72,9 @@ Unlike `aurora-postgres-server`, this service creates no AWS resources. It only
 
 | Workflow | Trigger | What It Does |
 |---|---|---|
-| `create` | Service created | Auto-discovers server, creates database + app user, writes service attributes |
+| `create` | Service created | Auto-discovers server, creates database + app user, stores app credentials in Secrets Manager, writes service attributes |
 | `update` | Service updated | No-op (no configurable parameters) |
-| `delete` | Service deleted | Reassigns owned objects to master, destroys app user; **database is preserved** |
+| `delete` | Service deleted | Reassigns owned objects to master, destroys app user and its Secrets Manager secret; **database is preserved** |
 | `link` | Application linked | Applies scoped grants for this link to the existing service-level PostgreSQL user |
 | `unlink` | Application unlinked | Revokes grants only; user and database are **preserved** |
 
@@ -85,10 +87,22 @@ username      = "app_<application_id>"
 
 This is a service-level derivation only — there is no separate per-link username. Every link on the same service shares this one PostgreSQL user, distinguished only by the grants each link's `access_level` applies to it.
 
+## Infrastructure Resources Created
+
+| Resource | Description |
+|---|---|
+| `postgresql_database` | The application database (preserved on delete) |
+| `postgresql_role` | The service-level app user |
+| `aws_secretsmanager_secret` | Stores the app-level credentials (`nullplatform/aurora/<service_id>/app`); destroyed alongside the app user on service delete |
+
+The app secret is encrypted with whatever KMS key the auto-discovered `aurora-postgres-server` has in its `secret_kms_key_id` attribute, so a cluster's master secret and all of its app secrets share one customer-managed key and one revocation point. When the server leaves it unset, AWS encrypts the app secret with the default `aws/secretsmanager` managed key. There is no per-database override: the key is an infrastructure-level choice that belongs on the server.
+
+No Aurora, EC2, or VPC resources are created — those belong to the auto-discovered `aurora-postgres-server`.
+
 ## Requirements
 
 - An active **`aurora-postgres-server`** service in the same nullplatform namespace with `status: active`, matching dimensions, and `hostname`/`master_secret_arn`/`engine_family` attributes already set.
-- See [`specs/install/README.md`](specs/install/README.md) and [`specs/requirements/aws`](specs/requirements/aws) for platform registration and the AssumeRole IAM role (selector `aurora-postgres-db`; Secrets Manager read access scoped to `nullplatform/aurora/*`).
+- See [`specs/install/README.md`](specs/install/README.md) and [`specs/requirements/aws`](specs/requirements/aws) for platform registration and the AssumeRole IAM role (selector `aurora-postgres-db`; Secrets Manager access — read the master secret, full lifecycle on the app secret it owns — scoped to `nullplatform/aurora/*`, plus `kms:Decrypt`/`kms:GenerateDataKey` scoped by `kms:ViaService` to Secrets Manager so a customer-managed `secret_kms_key_id` works). As with `rds-postgres-db`, this grant is shared per cluster, not per instance: anything that assumes the role can create/update/delete any secret under that prefix, not just its own.
 
 ## Important Considerations
 

aurora-postgres-db/db_setup/main.tf

@@ -34,3 +34,31 @@ resource "postgresql_role" "app_user" {
   password = random_password.user.result
   login    = true
 }
+
+# ---------------------------------------------------------------------------
+# App credentials, mirrored into Secrets Manager (same convention as the
+# aurora-postgres-server master secret) so they're not only reachable via
+# nullplatform service/link attributes.
+# ---------------------------------------------------------------------------
+
+resource "aws_secretsmanager_secret" "app" {
+  name                    = "nullplatform/aurora/${var.service_id}/app"
+  recovery_window_in_days = 0
+  kms_key_id              = var.secret_kms_key_id
+
+  tags = {
+    "managed-by" = "nullplatform"
+    "service-id" = var.service_id
+  }
+}
+
+resource "aws_secretsmanager_secret_version" "app" {
+  secret_id = aws_secretsmanager_secret.app.id
+  secret_string = jsonencode({
+    username = postgresql_role.app_user.name
+    password = random_password.user.result
+    host     = var.db_host
+    port     = var.db_port
+    dbname   = postgresql_database.app.name
+  })
+}

aurora-postgres-db/db_setup/outputs.tf

@@ -28,3 +28,8 @@ output "database_name" {
   value       = postgresql_database.app.name
   description = "Database name"
 }
+
+output "app_secret_arn" {
+  value       = aws_secretsmanager_secret.app.arn
+  description = "ARN of the Secrets Manager secret holding the app-level credentials"
+}

aurora-postgres-db/db_setup/providers.tf

@@ -8,9 +8,17 @@ terraform {
       source  = "hashicorp/random"
       version = "~> 3.0"
     }
+    aws = {
+      source  = "hashicorp/aws"
+      version = "~> 6.0"
+    }
   }
 }
 
+provider "aws" {
+  region = var.region
+}
+
 provider "postgresql" {
   host      = var.db_host
   port      = var.db_port

aurora-postgres-db/db_setup/variables.tf

@@ -3,6 +3,12 @@ variable "service_id" {
   description = "Nullplatform service ID (used as keeper to stabilize password across re-applies)"
 }
 
+variable "region" {
+  type        = string
+  default     = "us-east-1"
+  description = "AWS region (used to create the app credentials secret in Secrets Manager)"
+}
+
 variable "db_host" {
   type        = string
   description = "RDS endpoint hostname"
@@ -39,3 +45,9 @@ variable "master_password" {
   sensitive   = true
   description = "Master password for connecting to PostgreSQL (passed via auto.tfvars)"
 }
+
+variable "secret_kms_key_id" {
+  type        = string
+  default     = null
+  description = "KMS key ID or ARN used to encrypt the app credentials secret in Secrets Manager. Defaults to the aurora-postgres-server's secret_kms_key_id; if neither is set, AWS encrypts it with the default aws/secretsmanager managed key."
+}

aurora-postgres-db/scripts/aws/build_db_setup_context

@@ -33,6 +33,11 @@ fi
 SERVICE_ID=$(echo "$CONTEXT" | jq -r '.service.id')
 ACTION_TYPE=$(echo "$CONTEXT" | jq -r '.type // ""')
 
+# Inherited from the discovered aurora-postgres-server on create; stays empty on
+# delete, where the app secret is only destroyed and its encryption key is
+# already recorded in state.
+SECRET_KMS_KEY_ID=""
+
 # --- Resolve connection info -----------------------------------------------
 
 if [ -n "${SERVER_HOSTNAME:-}" ]; then
@@ -95,6 +100,17 @@ else
   DB_PORT=$(echo "$SERVER_JSON" | jq -r '.attributes.port // "5432"')
   MASTER_SECRET_ARN=$(echo "$SERVER_JSON" | jq -r '.attributes.master_secret_arn // ""')
 
+  # Encrypt the app secret with the same customer-managed key the server uses
+  # for its master secret, so both secrets of a cluster share one key and one
+  # revocation point. Unset on the server means the AWS-managed
+  # aws/secretsmanager key, same as before.
+  SECRET_KMS_KEY_ID=$(echo "$SERVER_JSON" | jq -r '.attributes.secret_kms_key_id // ""')
+
+  if [ -n "$SECRET_KMS_KEY_ID" ] && [[ ! "$SECRET_KMS_KEY_ID" =~ ^[A-Za-z0-9:/_-]+$ ]]; then
+    echo "ERROR: server ${SERVER_SERVICE_ID} has an invalid secret_kms_key_id '${SECRET_KMS_KEY_ID}' (expected a KMS key ID, ARN, or alias, with no whitespace)" >&2
+    exit 1
+  fi
+
   if [ -z "$DB_HOST" ]; then
     echo "ERROR: Server ${SERVER_SERVICE_ID} has no hostname attribute." >&2
     echo "       Has the aurora-postgres-server been created successfully?" >&2
@@ -146,4 +162,14 @@ export TOFU_IMPORT_DB_NAME="$DB_NAME_VAL"
 
 export TOFU_INIT_VARIABLES="-backend-config=bucket=${TFSTATE_BUCKET} -backend-config=key=db_setup.tfstate -backend-config=region=${REGION}"
 
-export TOFU_VARIABLES="-var=service_id=${SERVICE_ID} -var=db_host=${DB_HOST} -var=db_port=${DB_PORT} -var=db_name=${DB_NAME_VAL} -var=db_username=${DB_USERNAME_VAL} -var=master_username=${MASTER_USER} -var=master_secret_arn=${MASTER_SECRET_ARN}"
+TOFU_VARIABLES="-var=service_id=${SERVICE_ID} -var=region=${REGION} -var=db_host=${DB_HOST} -var=db_port=${DB_PORT} -var=db_name=${DB_NAME_VAL} -var=db_username=${DB_USERNAME_VAL} -var=master_username=${MASTER_USER} -var=master_secret_arn=${MASTER_SECRET_ARN}"
+
+# Omit the -var entirely when unset, rather than passing an empty string —
+# the Terraform variable defaults to null, which is what makes AWS fall back
+# to the aws/secretsmanager managed key. A "" value would instead fail as an
+# invalid KMS key ID.
+if [ -n "$SECRET_KMS_KEY_ID" ]; then
+  TOFU_VARIABLES="${TOFU_VARIABLES} -var=secret_kms_key_id=${SECRET_KMS_KEY_ID}"
+fi
+
+export TOFU_VARIABLES

aurora-postgres-db/scripts/aws/write_link_outputs

@@ -25,6 +25,7 @@ USERNAME=$(echo "$SERVICE_JSON"         | jq -r '.attributes.username          /
 PASSWORD=$(echo "$SERVICE_JSON"         | jq -r '.attributes.password          // ""')
 DATABASE_NAME=$(echo "$SERVICE_JSON"    | jq -r '.attributes.database_name     // ""')
 MASTER_SECRET_ARN=$(echo "$SERVICE_JSON"| jq -r '.attributes.master_secret_arn // ""')
+APP_SECRET_ARN=$(echo "$SERVICE_JSON"   | jq -r '.attributes.app_secret_arn    // ""')
 
 if [ -z "$USERNAME" ]; then
   echo "WARNING: No username found in service attributes. Skipping link attribute update."
@@ -38,13 +39,15 @@ ATTRS=$(jq -n \
   --arg password          "$PASSWORD" \
   --arg database_name     "$DATABASE_NAME" \
   --arg master_secret_arn "$MASTER_SECRET_ARN" \
+  --arg app_secret_arn    "$APP_SECRET_ARN" \
   '{
     hostname:          $hostname,
     port:              ($port | tonumber),
     username:          $username,
     password:          $password,
     database_name:     $database_name,
-    master_secret_arn: $master_secret_arn
+    master_secret_arn: $master_secret_arn,
+    app_secret_arn:    $app_secret_arn
   }')
 
 echo "Updating link ${LINK_ID_VAL} attributes:"

aurora-postgres-db/scripts/aws/write_service_outputs

@@ -23,6 +23,7 @@ USERNAME=$(echo "$ALL_OUTPUTS"         | jq -r '.db_username.value       // ""')
 PASSWORD=$(echo "$ALL_OUTPUTS"         | jq -r '.db_password.value       // ""')
 DATABASE_NAME=$(echo "$ALL_OUTPUTS"    | jq -r '.database_name.value     // ""')
 MASTER_SECRET_ARN=$(echo "$ALL_OUTPUTS"| jq -r '.master_secret_arn.value // ""')
+APP_SECRET_ARN=$(echo "$ALL_OUTPUTS"   | jq -r '.app_secret_arn.value    // ""')
 
 if [ -z "$HOSTNAME" ]; then
   echo "WARNING: No hostname output found. Skipping service attribute update."
@@ -36,21 +37,24 @@ ATTRS=$(jq -n \
   --arg password          "$PASSWORD" \
   --arg database_name     "$DATABASE_NAME" \
   --arg master_secret_arn "$MASTER_SECRET_ARN" \
+  --arg app_secret_arn    "$APP_SECRET_ARN" \
   '{
     hostname:          $hostname,
     port:              ($port | tonumber),
     username:          $username,
     password:          $password,
     database_name:     $database_name,
-    master_secret_arn: $master_secret_arn
+    master_secret_arn: $master_secret_arn,
+    app_secret_arn:    $app_secret_arn
   }')
 
 echo "Updating service ${SERVICE_ID} attributes:"
-echo "  hostname:      $HOSTNAME"
-echo "  port:          $PORT"
-echo "  username:      $USERNAME"
-echo "  database_name: $DATABASE_NAME"
-echo "  password:      ****"
+echo "  hostname:       $HOSTNAME"
+echo "  port:           $PORT"
+echo "  username:       $USERNAME"
+echo "  database_name:  $DATABASE_NAME"
+echo "  app_secret_arn: $APP_SECRET_ARN"
+echo "  password:       ****"
 
 np service patch --id "$SERVICE_ID" --body "{\"attributes\": $ATTRS}"
 echo "Service attributes updated successfully."

aurora-postgres-db/specs/links/connect.json.tpl

@@ -76,6 +76,13 @@
           "visibleOn": [],
           "editableOn": [],
           "description": "ARN of the Secrets Manager secret for master credentials (internal use)"
+        },
+        "app_secret_arn": {
+          "type": "string",
+          "export": false,
+          "visibleOn": [],
+          "editableOn": [],
+          "description": "ARN of the Secrets Manager secret holding the app-level credentials (internal use)"
         }
       }
     },

aurora-postgres-db/specs/requirements/aws/main.tf

@@ -21,22 +21,58 @@ resource "aws_iam_role" "nullplatform_aurora_postgres_db" {
 }
 
 ################################################################################
-# Secrets Manager IAM policy — read-only access to the Aurora master password
+# Secrets Manager IAM policy — read the Aurora master password, manage the
+# app-level credentials secret this service creates in db_setup/
+#
+# The KMS statement covers the case where the aurora-postgres-server sets
+# secret_kms_key_id: Secrets Manager then calls KMS on this role's behalf to
+# wrap and unwrap the secret, so without it CreateSecret/GetSecretValue fail
+# with AccessDenied on the key rather than on the secret. The key ARN is
+# operator-supplied and therefore not knowable here, so the grant is scoped by
+# kms:ViaService instead — these actions are only allowed when the call comes
+# through Secrets Manager, never for decrypting anything else with that key.
 ################################################################################
 
 resource "aws_iam_policy" "nullplatform_aurora_postgres_db_secretsmanager_policy" {
   count = local.iam_create ? 1 : 0
 
   name        = "${local.policies_name_prefix}-aurora-postgres-db-secretsmanager-policy"
-  description = "Policy for reading the Aurora master password from Secrets Manager"
+  description = "Policy for reading the Aurora master password and managing the app-level credentials secret in Secrets Manager"
 
   policy = jsonencode({
     Version = "2012-10-17"
-    Statement = [{
-      Effect   = "Allow"
-      Action   = "secretsmanager:GetSecretValue"
-      Resource = "arn:aws:secretsmanager:*:${data.aws_caller_identity.current.account_id}:secret:nullplatform/aurora/*"
-    }]
+    Statement = [
+      {
+        Effect = "Allow"
+        Action = [
+          "secretsmanager:CreateSecret",
+          "secretsmanager:DeleteSecret",
+          "secretsmanager:DescribeSecret",
+          "secretsmanager:GetSecretValue",
+          "secretsmanager:PutSecretValue",
+          "secretsmanager:UpdateSecret",
+          "secretsmanager:TagResource",
+          "secretsmanager:UntagResource",
+          "secretsmanager:GetResourcePolicy",
+          "secretsmanager:ListSecretVersionIds"
+        ]
+        Resource = "arn:aws:secretsmanager:*:${data.aws_caller_identity.current.account_id}:secret:nullplatform/aurora/*"
+      },
+      {
+        Effect = "Allow"
+        Action = [
+          "kms:Decrypt",
+          "kms:DescribeKey",
+          "kms:GenerateDataKey"
+        ]
+        Resource = "*"
+        Condition = {
+          StringLike = {
+            "kms:ViaService" = "secretsmanager.*.amazonaws.com"
+          }
+        }
+      }
+    ]
   })
 }
 

aurora-postgres-db/specs/requirements/aws/output.tf

@@ -14,7 +14,7 @@ output "permissions_role_id" {
 }
 
 output "secretsmanager_policy_arn" {
-  description = "ARN of the Secrets Manager read policy"
+  description = "ARN of the Secrets Manager policy (read master secret, manage app credentials secret)"
   value       = local.iam_create ? aws_iam_policy.nullplatform_aurora_postgres_db_secretsmanager_policy[0].arn : ""
 }
 

aurora-postgres-db/specs/service-spec.json.tpl

@@ -64,6 +64,13 @@
           "editableOn": [],
           "description": "ARN of the Secrets Manager secret for master credentials (internal use)",
           "order": 6
+        },
+        "app_secret_arn": {
+          "type": "string",
+          "visibleOn": [],
+          "editableOn": [],
+          "description": "ARN of the Secrets Manager secret holding the app-level credentials (internal use)",
+          "order": 7
         }
       }
     },

aurora-postgres-db/workflows/aws/delete.yaml

@@ -55,9 +55,10 @@ steps:
     file: $SERVICE_PATH/scripts/aws/do_tofu
     configuration:
       TOFU_ACTION: destroy
-      # Destroy only the user. The database is preserved with master as owner
-      # so data remains available for potential future use.
-      TOFU_TARGETS: "postgresql_role.app_user,random_password.user"
+      # Destroy only the user and its Secrets Manager secret. The database is
+      # preserved with master as owner so data remains available for
+      # potential future use.
+      TOFU_TARGETS: "postgresql_role.app_user,random_password.user,aws_secretsmanager_secret_version.app,aws_secretsmanager_secret.app"
 
   - name: cleanup tfstate bucket
     type: script