enclave

A Nix toolchain and Go runtime for running an application inside an AWS Nitro Enclave. The runtime establishes encrypted state under a KMS key whose policy is conditioned on the enclave's PCR0 measurement, exposes an attested HTTPS endpoint, and implements a blue/green migration protocol that transfers that state to a successor enclave with a different measurement.

The repository exports lib.buildEif for constructing enclave images, packages the runtime and client CLI, provides a development shell, and includes NixOS tests that exercise the runtime against an AWS emulator under nested KVM.

The runtime, enclave images, and checks support x86_64-linux. The CLI and development shell additionally support aarch64-linux and aarch64-darwin.

Repository layout

Path	Contents
`runtime/`	The Go runtime that runs as PID 1 inside the enclave. Separate Go module, `github.com/ArkLabsHQ/enclave/runtime`.
`client/`	Go client library for attestation-verified requests. Part of the root module.
`cmd/enclave/`	The `enclave` CLI.
`nix/`	`buildEif` function
`nix/tests/`	EIF construction and full blue/green runtime checks.

Quickstart

Add the flake as an input and build an enclave image from your application derivation.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    enclave.url = "github:ArkLabsHQ/enclave";
  };

  outputs =
    { nixpkgs, enclave, ... }:
    let
      system = "x86_64-linux";
      pkgs = import nixpkgs { inherit system; };

      myapp = pkgs.buildGoModule {
        pname = "myapp";
        version = "1.0.0";
        src = ./.;
        vendorHash = null;
      };

      eif = enclave.lib.buildEif {
        inherit pkgs;
        app = myapp;
        env = {
          ENCLAVE_DEPLOYMENT = "prod";
          ENCLAVE_APP_NAME = "myapp";
          ENCLAVE_AWS_REGION = "eu-west-1";
          ENCLAVE_PREVIOUS_PCR0 = "genesis";
        };
      };
    in
    {
      packages.${system} = {
        inherit eif;
      };
    };
}

Build the image and read its measurement:

nix build .#eif
cat result/pcr.json          # {"PCR0":"...","PCR1":"...","PCR2":"..."}

PCR0 is the identity of the enclave. It is what the KMS key policy is conditioned on and what clients pin. It changes whenever the runtime, the application, or the baked environment changes.

Provide the resulting EIF to a Nitro-capable host that satisfies the deployment requirements. Once it is running, verify it from a client:

nix run github:ArkLabsHQ/enclave -- curl /enclave/v1/info \
  --base-url https://enclave.example.com \
  --expected-pcr0 "$(jq -r .PCR0 result/pcr.json)"

Architecture

buildEif combines the packaged runtime, the application executable, and the baked environment into one measured EIF.

┌─ Enclave image, EIF ──────────────────────────────────────┐
│  /app/runtime   PID 1: clock, network, AWS, state, TLS    │
│  /app/<name>    application, exec'd by the runtime        │
│                 listens on 127.0.0.1:7074                 │
└───────────────────────────────────────────────────────────┘
          │ AWS APIs through host networking and IMDS
          │ HTTPS application and attestation endpoint
          ▼
   encrypted AWS state                 verified clients

The host launcher and infrastructure are external to this flake. Their required interfaces are documented under Deployment.

Ports

Endpoint	Direction	Purpose
vsock CID 3:1024	enclave to host	gvproxy L2 network
vsock CID 3:8002	enclave to host	IMDS forwarding
vsock CID 3:9000	EIF init to host	boot heartbeat
vsock :8003	host to enclave	migration control HTTP
TCP :443	public to enclave	TLS, runtime API, application proxy
TCP 127.0.0.1:8080	inside enclave	internal runtime API
TCP 127.0.0.1:7074	inside enclave	the application

The migration control API has no application-level authentication. The host must expose vsock port 8003 only through a restricted operator control plane.

State model

The runtime does not persist anything to disk. All state lives in AWS, encrypted under a KMS key that only the measured enclave can use.

Every key the runtime creates admits exactly one PCR0, whether it is a genesis key or the key a predecessor mints for its successor. Every Decrypt and GenerateDataKey call is attested, so a different enclave image cannot read the state even with the same IAM role.
A 32-byte storage DEK and each configured static secret are generated by attested GenerateDataKey calls and stored in SSM as key-scoped ciphertexts.
Each static secret is committed to a PCR: secret i extends PCR(16+i), which is then locked. The secrets are therefore part of the enclave's measurement from the point of generation onward.
/<deployment>/<app>/<locked|unlocked>/KMSKeyID/<pcr0> is written last, both at genesis and at migration finalisation. It is the atomic commit point for the enclave measuring <pcr0>: its value selects which generation of ciphertexts that enclave sees. It must never be managed by deployment tooling.
Because the pointer is PCR0-scoped, a handoff writes only into the successor's scope. The predecessor's own pointer, key, and ciphertexts are never touched, so it stays able to serve and to reboot for as long as you keep it.
At genesis, an immutable, attested deployment-genesis object naming the creating enclave's PCR0 is the final commit record. Before writing it, genesis claims its own KMSKeyID/<pcr0> with a create-only write.
Genesis is not a migration intent. It shares the bucket with the migration intent log and nothing else: it is deployment-wide rather than scoped to one PCR0, it never forms a sequence, and it is written once and never revised.

Boot paths

Whether a deployment already exists is decided by the Object-Locked deployment-genesis object, not by SSM alone. That key is fixed and identity-independent, so any enclave's genesis vetoes every later one. The preceding create-only SSM write prevents two enclaves from claiming different keys even if a lease expires between verification and commit.

Condition	Path	Behaviour
genesis object absent and `KMSKeyID/<pcr0>` absent	genesis	Requires no predecessor artifacts. Creates the key and snapshot, writes the receipt, claims `KMSKeyID/<pcr0>` without overwrite, then conditionally creates the immutable genesis object.
`KMSKeyID/<pcr0>` present and a state-origin receipt exists for this PCR0	resume	Verifies its own receipt, decrypts state, writes nothing.
`KMSKeyID/<pcr0>` present, no receipt for this PCR0, but a migration transition receipt and predecessor artifacts exist	adopt	Verifies the predecessor's attestation, the PCR31 commitment to its own PCR0, the KMS key policy, the transition receipt, the predecessor's migration intent, and last that the predecessor is the one `ENCLAVE_PREVIOUS_PCR0` committed to in the EIF — all before decrypting. Then writes its own state-origin receipt.
genesis object present and `KMSKeyID/<pcr0>` absent	fatal	The committed key claim was deleted; recovery is deliberately not automatic.
genesis object absent and `KMSKeyID/<pcr0>` present	fatal	Genesis was interrupted after claiming its key but before its final immutable commit.

Boot order is fixed and every step is fatal: clock synchronisation against /dev/ptp0, networking, AWS clients, telemetry, HTTP servers, state establishment, PCR extension, migration control server, TLS, SSM environment overlay, static secret export, then exec of the application.

Nix API

The flake exports one function under lib.

`lib.buildEif`

Builds a measured enclave image.

Arguments:

{
  pkgs,                  # x86_64-linux package set
  app,                   # package; executable selected with pkgs.lib.getExe
  env,                   # environment baked into the measurement
  extraPackages ? [ ],   # additional packages in the enclave rootfs
}

Produces a derivation containing image.eif and pcr.json.

buildEif selects the executable with pkgs.lib.getExe. Set app.meta.mainProgram when it differs from the package name. The executable is copied under /app, and APP_BINARY_NAME is injected automatically.
env is part of the measurement. Changing any value changes PCR0. buildEif does not currently validate runtime configuration; missing or invalid required values fail when the EIF boots.
The rootfs contains the system CA store and nothing else by default. The runtime never shells out. Applications that need /bin/sh or other utilities must request them: extraPackages = [ pkgs.busybox ].

Packages

The flake also exposes these packages:

Package	Systems	Purpose
`runtime`	`x86_64-linux`	The runtime executable embedded by `buildEif`.
`cli`, `default`	Linux and Darwin	The `enclave` client CLI.

For example, nix run github:ArkLabsHQ/enclave runs the client CLI, and nix build github:ArkLabsHQ/enclave#runtime builds the standalone runtime.

Development shell

nix develop provides Go, gopls, formatting tools, and golangci-lint for working on this repository.

Runtime configuration

Configuration is supplied through the EIF environment, which is part of the measurement. A subset can be overridden at runtime from SSM.

Identity and security

Variable	Default	Purpose
`ENCLAVE_DEPLOYMENT`	none	Required. First SSM path segment.
`ENCLAVE_APP_NAME`	none	Required. Second SSM path segment.
`ENCLAVE_DEV`	`false`	Selects the whole security envelope. When `true`: COSE signature and certificate chain verification of attestation documents is disabled, the `kvm-clock` assertion is skipped, the KMS key policy keeps its root recovery principal and the SSM namespace segment is `unlocked`, the genesis and migration-intent Object Lock retentions become five minutes and ten minutes, the migration cooldown becomes two seconds, and the clock-sync poll drops from five minutes to five seconds. When `false`: verification on, `kvm-clock` required, key policy locked, both retentions ten years, cooldown 24 hours, unless `ENCLAVE_MIGRATION_COOLDOWN` overrides it. There is no
way to ask for any other combination. For local testing against emulated NSM only. See Security notes.
`ENCLAVE_PREVIOUS_PCR0`	empty	The predecessor this image may adopt state from, or the literal `genesis` for an image that only ever genesises.
`ENCLAVE_SECRETS_CONFIG`	empty	JSON array of managed static secrets. Schema below.
`ENCLAVE_AWS_REGION`	`us-east-1`	Region for all AWS SDK clients.

Listeners and application

Variable	Default	Purpose
`ENCLAVE_APP_PORT`	`7074`	Port the application listens on.
`ENCLAVE_UPSTREAM`	`auto`	Runtime-to-application HTTP version. `h1` pins HTTP/1.1, `h2c` pins HTTP/2 cleartext and is required for gRPC, `auto` matches the inbound request.
`ENCLAVE_FQDN`	`localhost`	Hostname for the TLS certificate.
`ENCLAVE_VIPROXY_ENABLED`	`true`	Set to `false` to disable the in-process IMDS forwarder.
`ENCLAVE_VIPROXY_IN_ADDRS`	`127.0.0.1:80`	IMDS forwarder listen address.
`ENCLAVE_VIPROXY_OUT_ADDRS`	`3:8002`	IMDS forwarder target, `CID:PORT` or `host:port`.
`APP_BINARY_NAME`	`app`	Set by `buildEif` from the selected executable. The runtime execs `/app/<value>`.

The external TLS listener (443), the internal loopback listener (8080) and the host vsock port gvproxy listens on (1024) are fixed. The last of those is hardcoded on the host side too, so a value only the enclave knew about would silently break networking.

Migration

The S3 Object Lock retention on each migration intent record is not configurable: ten years in production, ten minutes under ENCLAVE_DEV. An operator who could shorten it could wait out the Object Lock and roll back undetected, so the measured image settles it. The cooldown between /request-migration and /finalise-migration is 24 hours in production and two seconds under ENCLAVE_DEV, and is the one setting here an operator may override, with ENCLAVE_MIGRATION_COOLDOWN baked into the image.

A successor ignores any intent record that is not retained under compliance mode, and any whose retain-until date does not cover the configured retention. Governance mode is refused because a caller holding s3:BypassGovernanceRetention can delete such an object, so it proves nothing about what was published. The retention check allows for upload delay and for skew between the writer's clock and the LastModified S3 stamps, using a fixed security-profile budget: two minutes in development, ten in production. It has no environment-variable override.

Clock

Production fails the boot unless the system clock source is kvm-clock. Under ENCLAVE_DEV the assertion is skipped, because the QEMU harness boots without the paravirtualized clock.

The runtime hard-steps the clock onto the PTP hardware clock at startup, then runs a PI servo that corrects frequency drift. Offsets above 100 ms trigger another hard-step. /dev/ptp0 is mandatory; the boot fails without it.

Logging and tracing

Variable	Default	Purpose
`ENCLAVE_MIGRATION_COOLDOWN`	posture default	Overrides the wait between `/request-migration` and `/finalise-migration`. Unset leaves the `ENCLAVE_DEV` posture in charge: 24 hours in production, two seconds in dev. Must parse as a duration and must not be negative; an explicit `0s` disables the wait. EIF-baked, never read from the SSM overlay.
`ENCLAVE_LOG_SHIP_INTERVAL`	`10s`	Flush cadence for logs, spans and the metrics snapshot. Log and span batches also flush at 250 events, or at 1 MiB.
`ENCLAVE_LOG_RETENTION_DAYS`	`30`	Retention applied to created log groups.

Log groups are /enclave/<deployment>/<app>/logs, /enclave/<deployment>/<app>/traces and /enclave/<deployment>/<app>/metrics.

Events timestamped more than an hour from now, either direction, are dropped on arrival, as are events over 256 KiB. Both are enclave policy, stricter than AWS requires. The narrow timestamp window keeps normally produced batches well inside the 24-hour span PutLogEvents rejects wholesale. An application that deliberately ships backdated telemetry will lose it.

AWS endpoint overrides

AWS_ENDPOINT_URL_KMS, AWS_ENDPOINT_URL_SSM, AWS_ENDPOINT_URL_STS, AWS_ENDPOINT_URL_S3, and AWS_ENDPOINT_URL_LOGS override the corresponding service endpoints. Setting the S3 endpoint also forces path-style addressing. These exist for testing against an emulator.

Static secrets

ENCLAVE_SECRETS_CONFIG is a JSON array:

[
  { "name": "signing-key", "env_var": "SIGNING_KEY" }
]

Field	Meaning
`name`	SSM path segment for the ciphertext.
`env_var`	Environment variable set on the application process, containing 64 lowercase hex characters.

Each secret is a 32-byte value generated by an attested KMS GenerateDataKey call and must be a valid secp256k1 private key; the runtime derives a public key to compute the PCR extension and rejects invalid values.

Constraints:

name must not be StorageDEK and must be unique.
Order is significant. Secret i is committed to PCR(16+i). PCR31 is reserved for migration, so at most 15 secrets are supported.
Changing the array changes the measurement, and therefore PCR0.

SSM environment overlay

Parameters under /<deployment>/<app>/env/ are read at boot (non-recursively, with decryption) and exported into the application's environment. This allows configuration changes without rebuilding the image.

Seven names are refused, because they define the enclave's identity, its lineage or its security posture and can only be changed by rebuilding: ENCLAVE_DEPLOYMENT, ENCLAVE_APP_NAME, ENCLAVE_SECRETS_CONFIG, ENCLAVE_DEV, ENCLAVE_MIGRATION_COOLDOWN, ENCLAVE_VERIFY_CLOCK_SOURCE, ENCLAVE_PREVIOUS_PCR0. The lock posture and the intent retention left the list by ceasing to be configuration at all — ENCLAVE_DEV settles them.

Five TLS and ACME settings are read only from this overlay, never from the baked environment, because TLS is configured before the overlay is applied to the application:

Parameter under `/<deployment>/<app>/env/`	Purpose
`ENCLAVE_FQDN`	Certificate hostname.
`ENCLAVE_USE_ACME`	`true` switches from self-signed to ACME.
`ENCLAVE_ACME_DIRECTORY`	`letsencrypt-staging` or an `https://` directory URL.
`ENCLAVE_ACME_EMAIL`	ACME account contact.
`ENCLAVE_ACME_CA`	PEM CA bundle for a private ACME server.

The TLS key is generated at genesis, encrypted with KMS, and included in the state root. Renewed certificates reuse it. The certificate bucket stores the certificate and, when ACME is enabled, the ACME account key.

Application process environment

The runtime execs the application with the full runtime environment — including the SSM overlay and static secrets — plus:

Variable	Value
`PORT`	`ENCLAVE_APP_PORT`, default `7074`
`ENCLAVE_APP_PORT`	the same value
`ENCLAVE_PROXY_PORT`	the internal API port, default `8080`
`ENCLAVE_RUNTIME_TOKEN`	a 32-byte hex bearer token, regenerated each boot

ENCLAVE_RUNTIME_TOKEN authenticates the application to the runtime's telemetry ingest endpoints. stdout and stderr are inherited.

SSM parameters

With D = deployment, A = app name, L = locked or unlocked:

Path	Written by	Purpose
`/D/A/CertBucketName`	operator	Shared certificate and ACME account-key bucket.
`/D/A/LeaseBucketName`	operator	Ephemeral coordination lease bucket.
`/D/A/env/<NAME>`	operator	Environment overlay.
`/D/A/L/KMSKeyID/<pcr0>`	runtime	Atomic commit point for the enclave measuring `<pcr0>`. Never manage this with deployment tooling.
`/D/A/L/StorageDEK/Ciphertext/<keyID>`	runtime	Encrypted storage DEK.
`/D/A/L/TLSKey/Ciphertext/<keyID>`	runtime	Encrypted TLS key.
`/D/A/L/<secret>/Ciphertext/<keyID>`	runtime	Encrypted static secret.
`/D/A/StateOriginReceipt/<keyID>/<pcr0>`	runtime	Attested proof of which enclave established this state.
`/D/A/MigrationStateOriginReceipt/<keyID>/<pcr0>`	runtime	Predecessor's attestation over the successor's state. Written create-only.
`/D/A/MigrationPreviousPCR0/<pcr0>`	runtime	Predecessor PCR0, written by the predecessor into its successor's scope.
`/D/A/MigrationPreviousKMSKeyID/<pcr0>`	runtime	Predecessor KMS key ID, committed into the successor's state root.
`/D/A/MigrationPreviousPCR0Attestation/<pcr0>`	runtime	Predecessor attestation after PCR31 commitment, same scoping.

Every runtime-written path is scoped by a key ID, a PCR0, or both, so nothing is ever overwritten. Each handoff therefore adds a generation rather than replacing one; prune retired generations only once you are certain you will never boot their PCR0 again.

KMSKeyID/<pcr0> and both state-origin receipts are written create-only, so the storage layer refuses a replacement rather than relying on the writer to check first.

Each state-origin receipt includes the generation's KMS key ID and its predecessor's PCR0 and KMS key ID. The audit follows those attested links without loading the generations' ciphertexts.

HTTP API

External listener, TCP :443

TLS 1.2 minimum. HTTP and gRPC clients authenticate the enclave by verifying its PCRs and pinning the live TLS public key to the PublicKey hash in the attestation document. /enclave/* responses also carry permissive CORS headers, and an OPTIONS preflight to any path in that namespace is answered 204 by the runtime.

Method	Path	Auth	Purpose
GET	`/enclave/attestation?nonce=<40 hex>`	none	NSM attestation document, base64. The nonce is mandatory and echoed back. `user_data` is exactly 39 bytes: ASCII `sha256:` followed by the raw 32-byte SHA-256 of the TLS PublicKey.
GET	`/enclave/v1/info`	none	Version, PCR0, predecessor PCR0 and attestation, migration status, application status, and the ancestor-key audit: every ancestor generation's PCR0, KMS key ID, and whether that key still exists, is pending deletion, or is gone.
GET	`/health`	none	`{"status":"ready"}` once the application has been started, `{"status":"initializing"}` with status 503 before.
POST	`/enclave/v1/metrics`	bearer	OTLP protobuf metrics ingest, 1 MiB limit.
POST	`/enclave/v1/logs`	bearer	OTLP protobuf logs ingest, 1 MiB limit.
POST	`/enclave/v1/traces`	bearer	OTLP protobuf spans ingest, 1 MiB limit.
any	unmatched paths outside the `/enclave` namespace	none	Reverse-proxied to the application.

Telemetry is ingest-only. It ships to CloudWatch and is never read back through the runtime, so a compromised enclave has no history to serve.

Bearer endpoints expect Authorization: Bearer <ENCLAVE_RUNTIME_TOKEN>.

The complete /enclave namespace is reserved for runtime APIs. Unknown non-preflight paths beneath it return the runtime's 404 response, and no request in that namespace is proxied to the application. A request to the bare /enclave is redirected to /enclave/, which then returns that 404.

/health reports ready as soon as the application process has been started, which is marginally before it binds its port. Readiness probes should target an application endpoint.

Ancestor-key audit

Each generation mints its own KMS key and migration is strictly additive, so a retired generation keeps a key that can still decrypt state until an operator deletes it. The ancestry block of /enclave/v1/info reports what became of those keys, newest first.

complete is true only when every state-origin receipt verifies and the walk reaches a receipt with no predecessor. Missing, altered, malformed or cyclic state makes it false and reason explains the failure without exposing the underlying AWS error. The genesis generation is not repeated separately: it is the final verified state-origin receipt in the chain.

`state`	Meaning
`exists`	`DescribeKey` reports that the key is present and not scheduled for deletion.
`pending_deletion`	Deletion is scheduled.
`deleted`	KMS no longer knows the key. This generation can no longer decrypt anything.
`unknown`	The state could not be read. The cause is logged, not published: the AWS error names role ARNs and account IDs, and this endpoint is unauthenticated.

unknown never means deleted. Only KMS reporting the key as absent produces deleted, so a permissions failure or a KMS outage can never be read as proof that a retired generation was retired.

The block is a cached snapshot, rebuilt once at boot and then daily: checked_at is when it was built, and is null before the first probe completes. Reading it never calls SSM or KMS, so the endpoint cannot be made slow or unavailable by either.

The chain has no length cap. A corrupt chain still terminates: a repeated PCR0/key identity stops the walk and is reported through complete and reason.

Internal listener, TCP 127.0.0.1:8080

Serves /v1/metrics, /v1/logs, /v1/traces, and /health only, with the same handlers and authentication. The HTTP method selects metric, log, and trace ingest (POST) or readback (GET). This is the endpoint advertised to the application through ENCLAVE_PROXY_PORT. It does not serve /enclave/v1/info, the /enclave/* endpoints, or the application proxy.

Migration control, vsock :8003

The host must provide trusted operators with controlled access to this vsock listener. It has no application-level authentication and must not be exposed to untrusted networks.

Method	Path	Body	Purpose
POST	`/request-migration`	`{"action":"requested"\|"aborted","target_pcr0":"<96 hex>"}`	Records an attested, Object-Locked intent in S3. Returns migration status.
POST	`/finalise-migration`	`{"new_pcr0":"<96 hex>"}`	Performs the handoff and flips `KMSKeyID`.

Status codes: 425 while the cooldown is active, 409 if no matching intent exists or it was aborted, 503 if the intent store is unavailable, 400 for a malformed body.

Deployment

This flake builds the EIF but does not provision or configure its host or AWS resources. Any deployment system may be used if it supplies the following interfaces.

Host requirements

Launch the EIF with AWS Nitro Enclaves and allocate sufficient CPU and memory.
Make /dev/nsm and /dev/ptp0 available inside the enclave.
Run gvproxy at host CID 3, vsock port 1024, with outbound connectivity to the configured AWS endpoints and any application dependencies.
Forward IMDS from host CID 3, vsock port 8002, to the host's instance metadata service so the runtime can obtain AWS credentials.
Answer the EIF boot heartbeat at host CID 3, vsock port 9000.
Expose the enclave's migration control listener on vsock port 8003 only to trusted operators.
Route intended client traffic to the enclave's TLS listener, TCP port 443 by default.

AWS requirements

Create a private S3 bucket for shared certificate state and write its name to /<deployment>/<app>/CertBucketName. Create a separate private S3 bucket for ephemeral coordination leases and write its name to /<deployment>/<app>/LeaseBucketName.

The migration intent bucket is not configured. Its name is derived, so no parameter a host can rewrite decides where deployment state is looked for:

enclave-<account-id>-<sha256(deployment \x00 app)[:8]>-migration-intents

Provisioning must create exactly that bucket, with versioning and Object Lock enabled at creation, before the enclave first boots; the runtime only reads and writes it. The digest keeps its name inside S3's 63-character limit and its character rules whatever the deployment and application are called, and the account ID keeps two AWS accounts off the same globally unique name.

AWS credentials delivered through IMDS must allow:

Statement	Permissions
`S3CertAndLeaseReadWrite`	`GetObject`, `PutObject`, `DeleteObject`, `ListBucket`, `GetBucketLocation` on the certificate and lease buckets.
`S3MigrationIntentObjectLock`	`PutObject`, `GetObject`, `GetObjectVersion`, `PutObjectRetention`, `ListBucket`, `ListBucketVersions`, `GetBucketLocation` on the derived intent bucket. Grant no `s3:CreateBucket`: the runtime must never manufacture an empty authority.
`SSMParams`	`GetParameter`, `GetParametersByPath`, `PutParameter` on `/<deployment>/<app>/*`.
`KMSAccess`	`CreateKey`, `TagResource`, `DescribeKey`. Locked keys also authorise `DescribeKey` through their `EnclaveOperations` statement.
`STSAccess`	`GetCallerIdentity`.
`CloudWatchLogsAccess`	Required, and write-only: `CreateLogGroup`, `CreateLogStream`, `PutLogEvents` on `/enclave/*`. `PutRetentionPolicy` is optional but recommended — without it the boot still succeeds and log groups never expire. Nothing more — the runtime never reads its own telemetry back, and granting `FilterLogEvents` or `DescribeLogStreams` would hand a compromised enclave the history it was designed not to hold. Read the logs with operator or CI credentials instead. Without this statement the enclave does not boot.

Encrypt, Decrypt, and GenerateDataKey are deliberately absent. Those operations are authorised by the enclave-created key's own PCR0-conditioned policy, not by the host credentials, so possessing those credentials is not sufficient to read enclave state.

DescribeKey is read-only metadata and grants nothing over ciphertext. It exists so a running enclave can report whether its ancestors' keys have been deleted; host credentials still cannot read enclave state with it.

/<deployment>/<app>/<locked|unlocked>/KMSKeyID/<pcr0> is owned exclusively by the runtime. Do not pre-create or declaratively manage it. Genesis claims it create-only, immediately before writing the deployment-genesis object; migration finalisation writes it last, as the atomic commit. A pre-existing value makes the runtime refuse to finalise a handoff onto that PCR0.

Blue/green migration

Migration transfers state from a running enclave to a successor with a different PCR0. The successor must not boot before the predecessor has finalised: it would find no artifacts in its own PCR0 scope, and fail rather than start fresh.

Nothing here is reversible and nothing needs to be. The predecessor keeps its own key, ciphertexts, and commit pointer throughout, so if the successor turns out to be bad you keep serving from the predecessor and never shift traffic. There is no rollback path because there is nothing to roll back.

The order is:

Build the successor EIF with ENCLAVE_PREVIOUS_PCR0 set to the predecessor's PCR0, and read its own PCR0. The value is measured into PCR0, so the successor's identity carries the predecessor it will accept.
Prepare the successor host and routing, but do not boot the successor.

Request the migration against the predecessor:

  curl -fsS -H 'Content-Type: application/json' \
    --data '{"action":"requested","target_pcr0":"<successor PCR0>"}' \
  http://<migration-control-endpoint>/request-migration

This writes an Object-Locked record to the intent log. It cannot be deleted.

Wait for the cooldown. Poll /enclave/v1/info until migration.state == "eligible".
Finalise:
```
  curl -fsS -H 'Content-Type: application/json' \
    --data '{"new_pcr0":"<successor PCR0>"}' \
  http://<migration-control-endpoint>/finalise-migration
```
The predecessor commits the successor's PCR0 into its own PCR31, creates a KMS key admitting the successor's PCR0 alone, re-encrypts the DEK and every static secret under it, writes its post-PCR31 attestation and the transition receipt, then writes KMSKeyID/<successor PCR0> last.

Finalising is not idempotent by design. If KMSKeyID/<successor PCR0> already holds a value the request is refused with 409, so a retry can never mint a second key and displace a generation the successor may already be running.

KMSKeyID/<successor PCR0> is written create-only, and that write is the handoff's commitment point. It is what makes the refusal above hold between independent enclaves rather than only within one process: if two predecessors sharing a PCR0 finalise the same intent concurrently, exactly one write succeeds and the other gets its 409. Everything written earlier in the step lives under a KMS key ID minted by that attempt alone, so a loser — or an attempt that dies partway — leaves only unreachable orphans: one KMS key and a few SSM parameters that nothing resolves. Retry is always safe, since the next attempt mints a fresh key and writes a disjoint set of paths.

An abort recorded after this write has committed does not retract the handoff. The intent log governs whether a migration may begin; the pointer is what makes it real.
Confirm KMSKeyID/<successor PCR0> now exists, and that KMSKeyID/<predecessor PCR0> is unchanged. The first is the commit; the second is the guarantee that the predecessor is still intact.
Boot the successor. It verifies the predecessor attestation, the PCR31 commitment, the key policy, the transition receipt, the predecessor's intent, and last that the predecessor named in SSM is the one its EIF committed to, before adopting the state.
Confirm adoption on the successor's /enclave/v1/info: previous_pcr0 equals the predecessor PCR0, previous_pcr0_attestation is non-empty, and migration.source_pcr0 equals the successor's own PCR0.
Shift client traffic using the deployment system's normal routing mechanism.
Keep both enclaves healthy for the soak period, then retire the predecessor. Its key and SSM generation stay live; leave them alone unless you are certain you will never boot that PCR0 again.
If you do retire a generation's key, schedule its deletion out of band, then poll the successor's /enclave/v1/info until that generation reports state: "deleted" in the ancestry block. Responses on that route are signed by the attestation-bound key, so that reading is the receipt that the retired generation can no longer decrypt anything. Preserve its state-origin receipt until the audit has verified the deletion; a missing receipt makes the ancestry incomplete rather than proving retirement.

Lock posture must not change across a handoff. ENCLAVE_DEV selects the locked/unlocked SSM namespace, so a successor that flips it looks in a different subtree, finds nothing, and fails to boot. A production image can therefore never adopt a deployment created by a dev image, or the reverse.

Upgrading across the PCR0-scoped receipt change

The transition receipt moved from MigrationStateOriginReceipt/<keyID> to MigrationStateOriginReceipt/<keyID>/<pcr0>. There is no fallback read, so predecessor and successor images must both carry the change, or neither. A handoff that straddles it writes the receipt where the successor will not look, and the successor fails to boot with predecessor artifacts present but no migration transition receipt.

This does not fail loudly on the predecessor: it commits KMSKeyID/<successor PCR0> and returns 200 before the successor ever reads. The refusal only appears when the successor boots, and a plain retry is then refused with 409 because the pointer is already committed.

Complete or abort in-flight migrations before upgrading. To recover a handoff that already straddled the change, delete the successor's committed pointer and finalise again:

aws ssm delete-parameter --name "/<D>/<A>/<locked|unlocked>/KMSKeyID/<successor PCR0>"

This is safe only while the successor has never booted — that pointer is the one thing standing between a successor and its state. The retry mints a fresh key and writes a disjoint generation; the abandoned one is orphaned, not reused.

Verifying an enclave

An enclave is only meaningful if clients verify it. Both the CLI and the library prove, before returning any response body, that they are talking to an enclave running the expected measured image.

CLI

nix run github:ArkLabsHQ/enclave -- curl <path> \
  --base-url <url> --expected-pcr0 <hex>

The Nix derivation is named enclave-cli; the installed binary is enclave.

Flag	Default	Purpose
`--base-url`	required	Enclave base URL.
`--expected-pcr0`	required	Expected PCR0, compared case-insensitively.
`-X`, `--method`	`GET`	HTTP method.
`-d`, `--data`	none	Request body. Sets `Content-Type: application/json`.
`-H`, `--header`	none	`Name: value`, repeatable.
`--strict-tls`	`false`	Additionally require public CA and hostname validation.
`--insecure-skip-cose-verify`	`false`	Skip COSE Sign1 + AWS Nitro root chain verification (QEMU/local test only; prints a warning). PCR0, nonce, the exact 39-byte TLS binding, and live certificate pinning are still checked.
`-v`, `--verbose`	`false`	Print request and verification summary to stderr.

# Runtime and migration status.
enclave curl /enclave/v1/info \
  --base-url https://enclave.example.com --expected-pcr0 834837d8...9ba9

# Authenticated POST.
enclave curl /v1/orders -X POST -d '{"amount":1000}' \
  -H "Authorization: Bearer $TOKEN" \
  --base-url https://enclave.example.com --expected-pcr0 834837d8...9ba9

The CLI exits non-zero if attestation or TLS pinning fails, and if the HTTP status is 400 or above.

Go client

import "github.com/ArkLabsHQ/enclave/client"

c, err := client.New("https://enclave.example.com", client.Options{
    ExpectedPCR0: "834837d8fdff29f35317acc40ba4e1e505b71a3cf7374ebba016a38e05c43784a01f0c1e88bf2b6174e4dbfc6f679ba9",
})
if err != nil {
    log.Fatal(err)
}

resp, err := c.Post(ctx, "/v1/orders", strings.NewReader(`{"amount":1000}`))
if err != nil {
    log.Fatal(err)
}

if resp.StatusCode >= 400 {
    log.Fatalf("HTTP %d: %s", resp.StatusCode, resp.Body)
}

Methods: Get, Post, Do, VerifyAttestation, GRPCConn. Package functions: New, NewFromManifest, PinnedHTTPClient, ManifestURL, FetchManifest.

Option	Default	Effect
`ExpectedPCR0`	required	`New` fails without it.
`ExpectedPCRs`	empty	Expected values for PCR16 onward, in order, matching `ENCLAVE_SECRETS_CONFIG`.
`CacheTTL`	`60s`	Attestation cache lifetime.
`StrictTLS`	`false`	Adds public CA and hostname validation on top of the attestation pin.
`InsecureSkipCOSEVerify`	`false`	Skips COSE signature and certificate chain verification. For local testing against emulated NSM only.
`InsecureTLS`	unset	Removes the certificate pin entirely.

What is verified on the first request, and cached for CacheTTL:

Check	Default	Relaxed by
Fresh 20-byte nonce echoed in the attestation document	always	nothing
COSE Sign1 signature and AWS Nitro root certificate chain	on	`InsecureSkipCOSEVerify`
PCR0 equals `ExpectedPCR0`	always	nothing
`user_data` is exactly `sha256:` plus the raw 32-byte TLS PublicKey SHA-256, and the live certificate contains that public key	on	`InsecureTLS`
Public CA and hostname validation	off	enabled by `StrictTLS`
PCR16 onward match `ExpectedPCRs`	off	populated by `ExpectedPCRs`

The certificate pin is installed from the attestation document before any request carrying data is made, so a request issued before verification completes fails closed.

GRPCConn uses the same PCR verification and attested TLS pinning model as HTTP, but does not perform public CA validation. Applications serving gRPC must set ENCLAVE_UPSTREAM=h2c.

Testing

nix flake check

Check	Purpose
`eif-build`	Builds predecessor and successor EIFs, validates PCR0 shape, and proves the measurements differ.
`e2e`	x86-only runtime lifecycle across ordinary `aws`, `blue`, and `green` NixOS nodes: direct AWS setup, genesis, clock recovery, attestation, ACME, migration, adoption, and restart recovery.

Unit tests are not flake checks. Run them with make test, or nix develop --command make test as CI does. make lint and make fmt are also available.

Requirements

Both checks run on x86_64-linux only. The e2e check uses QEMU's x86_64-only nitro-enclave machine type.

The e2e check needs a builder with:

/dev/kvm
nested virtualisation enabled
Nix system features kvm and nixos-test

Nested KVM is not optional. The runtime requires /dev/ptp0 inside the enclave, which the guest kernel provides through ptp_kvm, which in turn issues the KVM_HC_CLOCK_PAIRING hypercall. The intermediate VM's KVM only services that hypercall while its own clocksource is TSC-based. The blue and green test nodes therefore force clocksource=tsc; NixOS test instrumentation otherwise appends clocksource=acpi_pm, and the kernel honours the last value on the command line. Without this the enclave has no /dev/ptp0 and boot fails before networking.

After the cache is warm the full e2e test takes roughly four minutes.

Reading test output

--print-build-logs includes every test VM's kernel and systemd serial output. Filter it to keep evaluation progress, test driver actions, failures, and the result:

set -o pipefail
nix flake check --print-build-logs 2>&1 |
  rg --line-buffered \
    '(^evaluating |^checking |^error:|> (machine|aws|blue|green):|> !!!|> cleanup|> test script|all checks passed)'

pipefail preserves the exit status through the filter.

E2E boundaries

The e2e test uses three ordinary NixOS test nodes. aws runs the AWS emulator, the attestation-aware KMS Recipient proxy, IMDS, and ACME fixtures. blue and green launch measured EIFs with QEMU's nitro-enclave machine and vhost-device-vsock.

The test driver creates the required buckets and SSM parameters directly through AWS APIs, then controls node startup according to the runtime migration order. It does not simulate a deployment system, host image lifecycle, or traffic cutover.

The test EIF uses ENCLAVE_DEPLOYMENT=dev as its SSM namespace. It separately sets ENCLAVE_DEV=true because QEMU's emulated NSM produces no AWS certificate chain and the harness has no paravirtualized clock. That one flag also gives the suite the short Object Lock retentions and two-second cooldown it needs to exercise a handoff in seconds rather than years; it is only for local testing against the emulator.

Troubleshooting

The enclave does not start. Inspect the QEMU launcher and enclave console on the affected blue or green node.

starting clock sync failed: open /dev/ptp0. Nested KVM, invariant TSC exposure, or the clocksource. Confirm the guest's kernel command line ends with clocksource=tsc.

The runtime cannot obtain credentials. Confirm the test IMDS endpoint and the host's vsock port 8002 forward are reachable.

curl -fsS http://169.254.169.254/latest/meta-data/

/health is ready but application routes fail. /health only proves the application process started. Check an application endpoint directly and read the enclave console for application errors.

/request-migration returns an empty reply under QEMU. vhost-device-vsock 0.3 occasionally drops a forwarded host-to-guest connection before it reaches the enclave. Retry. This affects the emulated transport only; production uses Nitro AF_VSOCK. A genuine validation failure returns an HTTP status and body and should not be retried.

Security notes

ENCLAVE_DEV=true selects the insecure security envelope. It disables COSE signature verification, skips the kvm-clock assertion, leaves the KMS key policy amendable with its root recovery principal, and cuts both S3 Object Lock retentions to minutes and the migration cooldown to seconds — so a dev deployment's migration anchor can be waited out almost immediately. In that mode the runtime decodes attestation documents but does not verify their signature or validate the certificate chain against the AWS Nitro root. It logs INSECURE: skipping COSE signature verification of attestation document at startup. PCR comparison and user_data checks still apply. Only set ENCLAVE_DEV=true for local testing against emulated NSM. ENCLAVE_DEPLOYMENT is required but only selects the SSM namespace; values such as dev and prod do not control verification. Both settings are baked into the measurement and cannot be overridden from SSM.

CloudWatch is a hard boot dependency. Each stream is created and written to before the application starts, and a failure aborts the boot. The write matters: creating a log group proves nothing about being able to put events into it, so a role missing logs:PutLogEvents would otherwise boot clean and lose everything silently. An enclave whose telemetry goes nowhere cannot be audited, so it does not run. The trade is that a CloudWatch outage or a missing CloudWatchLogsAccess statement becomes an availability outage rather than a silent gap in the record.

Telemetry is not readable from the enclave. Logs and spans are shipped and forgotten; there is no queryable history and no endpoint that reads one back. The runtime's IAM statement is write-only for the same reason, so a host that compromises the enclave recovers neither a buffered window of application logs nor the ability to query what was already shipped.

Clients must pin PCR0. client.New refuses to construct a client without ExpectedPCR0. Without the pin, attestation proves only that some enclave is running, not that it is running your code.

HTTP and gRPC trust the attested TLS channel. The client verifies the Nitro attestation and PCRs before sending application requests, then pins the live TLS leaf to the exact hash carried in user_data.

Never manage KMSKeyID/<pcr0> with deployment tooling. Migration finalisation rewrites it as its atomic commit, and genesis claims it create-only. A declaratively managed value would fight the runtime and could roll a live deployment back to a key that no longer decrypts anything. Once the deployment-genesis object exists, deleting the parameter cannot fork the state — the boot fails instead of creating a second generation — but it still stops the deployment booting until it is restored. Genesis claims the parameter before writing that object, so a crash between the two leaves a window in which deleting the parameter does re-open genesis.

Genesis is committed exactly once. Every contender first performs a create-only SSM key claim. Only that winner writes the deployment-genesis object, itself a create-only put under Object Lock, so a second write is rejected rather than layered on top. Its attestation binds the creating enclave's PCR0 and the bucket. Boot lists that one key's versions rather than reading it directly, because Object Lock cannot stop a delete marker hiding the object, and it reads them unverified: a record it cannot verify still vetoes a genesis, because treating it as absent would fail open. The contents are therefore advisory — presence is the only thing boot trusts.

The intent bucket is on the boot path. An unreachable or unwritable bucket now fails the boot rather than only blocking migration. That is the intended trade: an enclave that cannot check whether the deployment exists must not guess.

Seed the genesis object before adopting this on any existing deployment. No runtime before this change wrote one, so every deployment created earlier lacks it — migrated or not. Create the derived bucket and write a deployment-genesis object naming the running PCR0 into it before upgrading. Without the bucket the boot fails reading the store; with an empty one it reaches the genesis path, finds its committed KMSKeyID/<pcr0>, and fails there.

Existing intent records cannot be carried across. Each record's bucket_name is part of its attested pre-image, so one copied into the derived bucket no longer verifies and is skipped without an error — migration history restarts. The seeded record is unverifiable for the same reason, since only an enclave of that PCR0 could sign one. That is enough for the genesis check, which is unverified by design, but the record never appears as a migration head, so the first real handoff is written at sequence 1 beside it.

Host credentials cannot read enclave state. They grant kms:CreateKey but not Decrypt, Encrypt, or GenerateDataKey. Those are authorised by the enclave-created key policy, which is conditioned on PCR0. Compromising the host does not yield the state.

Static secrets are measured. Each is committed to a PCR that is then locked, so a successor enclave cannot silently substitute a different value; migration carries the ciphertexts forward under a key admitting the successor's measurement alone.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
client		client
cmd/enclave		cmd/enclave
nix		nix
runtime		runtime
.gitignore		.gitignore
.golangci.yml		.golangci.yml
Makefile		Makefile
README.md		README.md
flake.lock		flake.lock
flake.nix		flake.nix
go.mod		go.mod
go.sum		go.sum

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

enclave

Contents

Repository layout

Quickstart

Architecture

Ports

State model

Boot paths

Nix API

lib.buildEif

Packages

Development shell

Runtime configuration

Identity and security

Listeners and application

Migration

Clock

Logging and tracing

AWS endpoint overrides

Static secrets

SSM environment overlay

Application process environment

SSM parameters

HTTP API

External listener, TCP :443

Ancestor-key audit

Internal listener, TCP 127.0.0.1:8080

Migration control, vsock :8003

Deployment

Host requirements

AWS requirements

Blue/green migration

Upgrading across the PCR0-scoped receipt change

Verifying an enclave

CLI

Go client

Testing

Requirements

Reading test output

E2E boundaries

Troubleshooting

Security notes

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`lib.buildEif`