diff --git a/docs/docs/decisions/0003-use-kms-with-pgp-recovery-for-secrets.md b/docs/docs/decisions/0003-use-kms-with-pgp-recovery-for-secrets.md new file mode 100644 index 0000000..a24877d --- /dev/null +++ b/docs/docs/decisions/0003-use-kms-with-pgp-recovery-for-secrets.md @@ -0,0 +1,124 @@ +--- +status: accepted +date: 2026-08-18 +--- + +# ADR-0003: Use AWS KMS with PGP Recovery for Secrets + +## Context and Problem Statement + +`GilmanLab/secrets` must remain usable during routine operation without +long-lived credentials and must also recover the lab if its AWS account or SOPS +KMS key is lost. The previous glab-era rule allowed only KMS recipients, making +the AWS account a single recovery dependency. What should protect SOPS data +keys for lab v2? + +## Decision Drivers + +- Use short-lived, scope-limited AWS credentials for normal operation. +- Preserve a physically controlled recovery path outside AWS. +- Keep access boundaries enforceable for automation principals. +- Retain recovery-critical generated secrets across a complete lab rebuild. +- Avoid changing existing encrypted-file paths or plaintext values. + +## Considered Options + +- Wrap every SOPS data key to AWS KMS and a YubiKey-backed PGP recipient in one + key group. +- Continue using AWS KMS as the only recipient. +- Put KMS and PGP in separate SOPS key groups. +- Move secret storage immediately to Vault. + +## Decision Outcome + +Every SOPS file uses one key group containing two alternative recipients: + +- the lab SOPS KMS key, with encryption context + `Repo=GilmanLab/secrets` and a scope-specific `Scope` value; and +- Josh's YubiKey-backed PGP identity as a break-glass recipient. + +SOPS targets the identity's Curve25519 encryption subkey explicitly. Selecting +the primary fingerprint allowed SOPS' OpenPGP implementation to choose the +Ed25519 authentication subkey, producing a packet the YubiKey could not +decrypt. The primary key remains the recovery identity; the configured +recipient is its encryption-capable subkey with exact-subkey selection. + +Recipients inside one SOPS key group are alternatives: either KMS or the +YubiKey can recover the data key. Separate groups invoke Shamir secret sharing +and would require both trust roots, which does not survive AWS loss. + +Routine automation uses KMS. PGP deliberately bypasses IAM scope boundaries; +its boundary is physical YubiKey custody. IAM roles live in `GilmanLab/aws` and +are created only for real consumers. Each role is restricted to one workflow, +the SOPS KMS key, and both encryption-context conditions: + +- `kms:EncryptionContext:Repo = GilmanLab/secrets` +- `kms:EncryptionContext:Scope = ` + +An unconditioned KMS grant spans every scope and is a defect. + +Non-generated secrets must live in `GilmanLab/secrets`. Generated but durable, +lab-recovery-critical secrets also live there as an exception, including +IncusOS ZFS recovery keys, the seed/bootstrap client private key, and the image +factory's cache-signing key. Ephemeral runtime-generated secrets are not +committed. + +### Consequences + +- Good, because ordinary access uses MFA-backed SSO or short-lived OIDC + credentials and can be limited to one scope. +- Good, because physical PGP recovery remains available after total AWS loss. +- Good, because recovery-critical generated material survives the systems that + normally consume it. +- Bad, because one physical recovery identity can bypass all IAM scope + boundaries. +- Bad, because YubiKey custody and periodic PGP-only recovery tests become + operational requirements. +- Bad, because recipient changes require rewrapping every affected SOPS data + key and verifying both paths. + +### Confirmation + +The implementation conforms when: + +- `.sops.yaml` in `GilmanLab/secrets` defines KMS and PGP in one key group for + every scope; +- every encrypted file carries the expected KMS ARN, `Repo` and `Scope` + context, and exact PGP encryption-subkey recipient; +- CI validates that metadata without AWS credentials or plaintext access; +- KMS-only decryption preserves the known plaintext hash for each file; +- a YubiKey decrypts at least one file per scope with AWS credentials absent; +- every consumer IAM policy conditions `kms:Decrypt` on both context values. + +## Pros and Cons of the Options + +### KMS and PGP in One Key Group + +- Good, because either trust root can recover a data key independently. +- Good, because KMS remains compatible with per-scope least privilege. +- Bad, because compromise of either trust root can decrypt affected files. + +### KMS Only + +- Good, because all access is centrally auditable and conditionable in IAM. +- Bad, because loss of the AWS account or KMS key destroys the only decryption + path. + +### Separate KMS and PGP Key Groups + +- Good, because decryption requires independent trust roots. +- Bad, because AWS loss prevents recovery, contradicting the primary goal. + +### Vault Now + +- Good, because Vault can issue dynamic credentials and centralize policy. +- Bad, because Vault depends on lab infrastructure and cannot be the root of + trust for secrets needed to build that infrastructure. + +## More Information + +The canonical recipient, path rules, encryption context, and operational +commands are in the +[`GilmanLab/secrets` README](https://github.com/GilmanLab/secrets/blob/master/README.md). +The KMS key is managed by the `aws/lab-foundation` root in `GilmanLab/aws`; +consumer IAM roles will be implemented there when their workflows exist. diff --git a/docs/docs/index.md b/docs/docs/index.md index 35ff96b..076ebc6 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -16,6 +16,9 @@ and runbook lives here. assigns core routing and switching responsibilities. - [ADR-0002: Manage the Tailnet Policy File with GitOps](decisions/0002-manage-tailscale-policy-with-gitops.md) makes git the source of truth for the tailnet policy file. +- [ADR-0003: Use AWS KMS with PGP Recovery for Secrets](decisions/0003-use-kms-with-pgp-recovery-for-secrets.md) + uses scoped KMS access for routine decryption and a YubiKey-backed PGP key + for recovery outside AWS. ## Designs diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index e1e1362..93e059f 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -34,6 +34,7 @@ nav: - Decisions: - "ADR-0001: Use VyOS for Layer 3 and Dedicated Switches for Layer 2": decisions/0001-use-vyos-for-layer-3-and-switches-for-layer-2.md - "ADR-0002: Manage the Tailnet Policy File with GitOps": decisions/0002-manage-tailscale-policy-with-gitops.md + - "ADR-0003: Use AWS KMS with PGP Recovery for Secrets": decisions/0003-use-kms-with-pgp-recovery-for-secrets.md - Designs: - Drafts: - Lab v2 core network: designs/drafts/lab-v2-core-network.md