diff --git a/.gitignore b/.gitignore index d0e3fa8..a312d5b 100644 --- a/.gitignore +++ b/.gitignore @@ -34,3 +34,7 @@ dist/ # IDE .vscode/ .idea/ + +# Generated judge-demo evidence (keep only the directory marker) +artifacts/demo-platform/* +!artifacts/demo-platform/.gitkeep diff --git a/BUILD_WEEK.md b/BUILD_WEEK.md index fe4a845..bb236dc 100644 --- a/BUILD_WEEK.md +++ b/BUILD_WEEK.md @@ -44,6 +44,60 @@ Work started during the submission period includes: | Phase 5 | Real-cluster agent and console lifecycle integration | Issue #101 / PR #102 | | Phase 6 | OpenAI-native reasoning, configuration, deployment, UI status, and documentation | Issue #103 / PR #104 | | Phase 7 | One-command deterministic Argus, Phoenix, Sentinel, and SOG judge demo | Issue #110 | +| Phase 8 | Auditable judge evidence scorecard and JSON/Markdown run artifacts | Issue #112 | +| Phase 9 | Cluster-free full-platform judge demo and explicit live-k3s command | Issue #114 | +| Phase 10 | Guarded live k3s detection, chaos, recovery, correlation, and measured evidence | Issue #116 | +| Phase 11 | Judge onboarding doctor, single-path setup, and rehearsal diagnostics | Issue #118 | +| Phase 12 | Canonical resilience lifecycle and unambiguous evidence provenance | Argus #120 / Sentinel #63 | +| Final video | Two-minute judge story with frame-by-frame screen, action, narration, timing, and fallback plan | Pending | + +### Phase 12 — truthful resilience proof timeline + +Every correlated judge incident can now carry the same seven-stage lifecycle: Healthy, +Fault injected, Detection, Decision, Human approval, Recovery, and Verification. The +live producer supplies actual timestamps, measured detection/recovery durations, HTTP +availability, experiment ID, and evidence source. Human approval occurs only after +observed Argus evidence exists. The portable producer supplies a visually equivalent +timeline, but every stage is explicitly simulator/replay provenance and availability is +reported as not measured. Sentinel renders four mutually exclusive labels—Live +Observed, Synthetic Simulator, Live Chaos Mesh, and Replayed Evidence—and never infers +missing proof stages. + +### Phase 11 — judge onboarding and rehearsal doctor + +`make doctor` and `make doctor-live` are read-only entry points for the portable and live +proofs. They validate the sibling repository layout, required tools, secret-safe OpenAI +configuration status, runtime availability, local console dependencies, and—only in +live mode—the complete k3s stack. Each run ends with an unambiguous verdict and exact +next command. `setup.md`, the root README, and Make help now present the same choose-one +path and distinguish replay/simulator claims from observed/live-chaos measurements. + +### Phase 10 — observed live-cluster proof + +`make demo-platform-live-dry-run` validates the actual k3s security, chaos, agent, and +SOG stack without changing it. `make demo-platform-live` then requires two explicit +human confirmations, creates an isolated two-replica service, waits for observed Argus +evidence, asks Phoenix to create one real Chaos Mesh PodChaos, continuously measures +HTTP availability, verifies Kubernetes replacement and full readiness, and publishes +only the verified cross-agent lifecycle to Sentinel. The resulting JSON/Markdown +scorecard distinguishes observed/live-chaos provenance from the portable simulator. + +### Phase 9 — portable judge demo + +`make demo-platform` now defaults to the cluster-free path: a disposable local SOG, +real local product services, an explicitly synthetic three-node topology, deterministic +Argus and Phoenix lifecycles, and a bounded dynamic feed. `make demo-platform-live` +preserves the k3s-backed integration proof. The split prevents “safe” from being confused +with “cluster-free” and keeps simulated evidence visually useful without mislabeling it +as live production telemetry. + +### Phase 8 — judge evidence report + +Every successful deterministic platform run now emits a concise PASS scorecard and +stores the verified lifecycle as JSON and Markdown. The report preserves correlation, +sources, provenance, seed, governance, OpenAI configuration, recovery state, and measured +orchestration timings. It explicitly avoids claiming live detection latency or 100% +availability from replayed and simulated evidence. ### Phase 7 — deterministic platform demo diff --git a/Makefile b/Makefile index 03709cf..e97f52d 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ .PHONY: help cluster-up cluster-down cluster-status deploy-cilium deploy-falco \ deploy-kyverno deploy-observability deploy-agent deploy-ui \ - setup-local dev-agent dev-ui demo-local demo-platform demo-platform-dry-run demo-cluster demo-cluster-dry-run hubble-ui grafana-ui k9s \ + setup-local doctor doctor-live dev-agent dev-ui demo-local demo-platform demo-platform-local demo-platform-live demo-platform-dry-run demo-platform-live-dry-run demo-cluster demo-cluster-dry-run hubble-ui grafana-ui k9s \ test test-agent test-ui test-cluster-demo test-platform-demo simulate-threats clean THREAT_COUNT ?= 10 @@ -27,10 +27,15 @@ help: @echo " make deploy-observability Install Prometheus + Grafana + Loki" @echo "" @echo " Application" + @echo " make doctor Diagnose the portable judge path without changes" + @echo " make doctor-live Diagnose the live k3s proof without changes" @echo " make setup-local Install local backend and UI dependencies" @echo " make demo-local Start a populated cluster-free demo" - @echo " make demo-platform Start the full Argus + Phoenix + Sentinel judge demo" - @echo " make demo-platform-dry-run Validate the full platform demo without starting anything" + @echo " make demo-platform Start the cluster-free full-platform judge demo" + @echo " make demo-platform-local Same cluster-free full-platform judge demo" + @echo " make demo-platform-live Start the real k3s-backed platform proof" + @echo " make demo-platform-dry-run Validate local demo prerequisites without changes" + @echo " make demo-platform-live-dry-run Validate live k3s prerequisites without changes" @echo " make dev-agent Start only the backend on localhost:8000" @echo " make dev-ui Start only the console on localhost:5173" @echo " make deploy-agent Build and deploy AI agent" @@ -104,6 +109,12 @@ setup-local: .venv/bin/pip install -r agent/requirements.txt npm --prefix ui ci +doctor: + @bash scripts/demo-doctor.sh local + +doctor-live: + @bash scripts/demo-doctor.sh live + dev-agent: @test -x .venv/bin/python || (echo "Missing local environment. Run: make setup-local" && exit 1) @cd agent/src && ../../.venv/bin/python -m uvicorn main:app --reload --host 127.0.0.1 --port 8000 @@ -118,12 +129,26 @@ demo-local: demo-platform: @PHOENIX_ROOT="$${PHOENIX_ROOT:-$(abspath ../sentinel-stack/phoenix)}" \ SENTINEL_ROOT="$${SENTINEL_ROOT:-$(abspath ../sentinel-stack/sentinel)}" \ - bash scripts/demo-platform.sh + SENTINEL_PLATFORM_ROOT="$${SENTINEL_PLATFORM_ROOT:-$(abspath ../sentinel-stack/sentinel-platform)}" \ + bash scripts/demo-platform-local.sh + +demo-platform-local: demo-platform + +demo-platform-live: + @PHOENIX_ROOT="$${PHOENIX_ROOT:-$(abspath ../sentinel-stack/phoenix)}" \ + SENTINEL_ROOT="$${SENTINEL_ROOT:-$(abspath ../sentinel-stack/sentinel)}" \ + bash scripts/demo-platform-live-proof.sh demo-platform-dry-run: @PHOENIX_ROOT="$${PHOENIX_ROOT:-$(abspath ../sentinel-stack/phoenix)}" \ SENTINEL_ROOT="$${SENTINEL_ROOT:-$(abspath ../sentinel-stack/sentinel)}" \ - DEMO_PLATFORM_DRY_RUN=true bash scripts/demo-platform.sh + SENTINEL_PLATFORM_ROOT="$${SENTINEL_PLATFORM_ROOT:-$(abspath ../sentinel-stack/sentinel-platform)}" \ + DEMO_PLATFORM_DRY_RUN=true bash scripts/demo-platform-local.sh + +demo-platform-live-dry-run: + @PHOENIX_ROOT="$${PHOENIX_ROOT:-$(abspath ../sentinel-stack/phoenix)}" \ + SENTINEL_ROOT="$${SENTINEL_ROOT:-$(abspath ../sentinel-stack/sentinel)}" \ + LIVE_DEMO_DRY_RUN=true bash scripts/demo-platform-live-proof.sh demo-cluster: @DEMO_CLUSTER_CONTEXT="$(DEMO_CLUSTER_CONTEXT)" \ @@ -147,6 +172,8 @@ test-ui: test-platform-demo: @bash scripts/tests/test-demo-platform.sh + @bash scripts/tests/test-demo-platform-live.sh + @bash scripts/tests/test-demo-doctor.sh test-cluster-demo: @bash scripts/tests/test-demo-cluster.sh diff --git a/README.md b/README.md index bf53839..129193f 100644 --- a/README.md +++ b/README.md @@ -54,7 +54,8 @@ cross-agent judge experience. |---|---|---| | **Local synthetic** | You want the fastest judge/reviewer experience and do not have Kubernetes | `make demo-local` | | **Live k3s** | You have the three-node Argus cluster and want real Falco, Cilium, and Kyverno evidence | `make demo-cluster-dry-run`, then `make demo-cluster` | -| **Full platform** | You want Argus, Phoenix, Sentinel, and the SOG operating as one deterministic demo | `make demo-platform-dry-run`, then `make demo-platform` | +| **Full platform, cluster-free** | Recommended judge path: all three products, realistic synthetic topology, no Kubernetes | `make demo-platform-dry-run`, then `make demo-platform` | +| **Full platform, live k3s** | Maintainer/video path: observed evidence, approved live chaos, measured recovery | `make doctor-live`, then `make demo-platform-live` | ### Path A — local synthetic, no cluster required @@ -132,19 +133,19 @@ injects the real workloads, prints terminal evidence, and keeps the console avai at **http://127.0.0.1:5173** until `Ctrl-C`. Cleanup stops both local processes and deletes only the namespace created by that run. -### Path C — full deterministic platform demo +### Path C — full cluster-free platform demo (recommended) -Use this path for the complete judge story. It requires sibling Phoenix and Sentinel -checkouts plus the deployed `sentinel-platform` and `phoenix-system` services in the -selected Kubernetes context. Validate everything without starting processes, opening -port-forwards, or publishing evidence: +Use this path for the complete judge story. It requires sibling Phoenix, Sentinel, and +Sentinel Platform checkouts plus Docker or OrbStack for one disposable Redis container. +It does **not** require Kubernetes, kubectl, k3s, Cilium, Falco, or Chaos Mesh. ```text Projects/ ├── argus-k8s/ # run the command here └── sentinel-stack/ ├── phoenix/ - └── sentinel/ + ├── sentinel/ + └── sentinel-platform/ ``` Install each repository's local dependencies once: @@ -155,11 +156,10 @@ make -C ../sentinel-stack/sentinel setup-local npm --prefix ../sentinel-stack/phoenix/dashboard install ``` -Select the real three-node k3s context, then run the non-mutating preflight: +Run the non-mutating preflight: ```bash -kubectl config use-context argus -make demo-platform-dry-run +make doctor ``` Then launch the complete experience: @@ -168,10 +168,20 @@ Then launch the complete experience: make demo-platform ``` -The command starts or reuses all required APIs, port-forwards, and dashboards; publishes -one deterministic Argus security finding and one Phoenix recovery outcome against the -same resource; and refuses to report success until Sentinel exposes their correlated -incident with explicit `replayed` and `simulator` provenance. +The command installs missing local dependencies, starts a disposable Redis-backed real +SOG and the real local Argus, Phoenix, and Sentinel services, then seeds a three-node, +multi-namespace service graph. It populates Argus with twelve threats, Phoenix with a +synthetic dependency graph, and Sentinel with multiple correlated lifecycles. A bounded +feed adds a new replay/simulator lifecycle every 20 seconds so refresh timestamps, +counters, timelines, and risk views visibly move during the demo. + +Every successful run prints a judge-readable PASS scorecard and writes the exact proof +to `artifacts/demo-platform/latest-demo.json` and +`artifacts/demo-platform/latest-demo.md`. The report includes evidence-publication and +correlation timings, recovery status, approval policy, OpenAI availability, sources, +seed, and provenance. It deliberately reports availability as **not measured** for this +deterministic path instead of turning a simulator recovery into a false production-SLA +claim. | Console | URL | What to show | |---|---|---| @@ -179,11 +189,34 @@ incident with explicit `replayed` and `simulator` provenance. | Phoenix | **http://127.0.0.1:5174** | Resilience outcome and recovery | | Sentinel | **http://127.0.0.1:5175** | Unified correlated incident and fleet decision | -Existing healthy services are reused. On `Ctrl-C`, the command stops only processes it -started. The default sibling paths are `../sentinel-stack/phoenix` and -`../sentinel-stack/sentinel`; override them with `PHOENIX_ROOT` and `SENTINEL_ROOT` when -your checkouts live elsewhere. This deterministic path does not inject live Chaos Mesh -faults; the existing live k3s path remains available for real runtime security evidence. +Every synthetic entity has `demo-data=synthetic`; evidence is explicitly labeled +`replayed` or `simulator`. The local demo never claims live Falco detection, live chaos, +or measured production availability. On `Ctrl-C`, it stops its local processes and +removes its disposable Redis container. It replaces only known project listeners and +kubectl port-forwards on reserved demo ports; unrelated listeners cause a safe failure. +The Sentinel incident drawer uses the same seven-stage resilience timeline as the live +proof, while labeling every portable stage as replayed or simulated and leaving +availability explicitly unmeasured. + +### Path D — live k3s-backed platform proof + +Run the guarded real-cluster proof separately: + +```bash +kubectl config use-context argus +make doctor-live +make demo-platform-live +``` + +The dry-run is read-only. The real command verifies Cilium, Falco, Kyverno, Argus, +Phoenix, Chaos Mesh, and the SOG; asks for the exact Kubernetes context and the phrase +`INJECT LIVE FAULT`; creates only `sentinel-live-demo`; and launches a two-replica HTTP +service. Argus must observe a bounded Falco-triggering workload before Phoenix creates a +real Chaos Mesh `PodChaos` against one disposable replica. The proof passes only after a +new replacement pod is Ready, both replicas are Ready, continuous HTTP availability is +measured, and Sentinel exposes the correlated Argus + Phoenix incident. `Ctrl-C` stops +the consoles and deletes only the isolated demo namespace. Evidence is written to +`artifacts/demo-platform/latest-live-demo.{json,md}`. ## Part of the Sentinel multi-agent platform diff --git a/artifacts/demo-platform/.gitkeep b/artifacts/demo-platform/.gitkeep new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/artifacts/demo-platform/.gitkeep @@ -0,0 +1 @@ + diff --git a/scripts/demo-doctor.sh b/scripts/demo-doctor.sh new file mode 100644 index 0000000..f115269 --- /dev/null +++ b/scripts/demo-doctor.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +mode="${1:-local}" +phoenix_root="${PHOENIX_ROOT:-${repo_root}/../sentinel-stack/phoenix}" +sentinel_root="${SENTINEL_ROOT:-${repo_root}/../sentinel-stack/sentinel}" +platform_root="${SENTINEL_PLATFORM_ROOT:-${repo_root}/../sentinel-stack/sentinel-platform}" +failures=0 +warnings=0 + +usage() { + cat <<'EOF' +Usage: scripts/demo-doctor.sh [local|live] + + local Check the recommended cluster-free judge path (default). + live Check the observed k3s proof, including deployed stack readiness. + +This command is read-only. It never starts services, installs dependencies, publishes +evidence, creates Kubernetes resources, or injects a fault. +EOF +} + +pass() { printf ' [PASS] %s\n' "$1"; } +warn() { printf ' [WARN] %s\n' "$1"; warnings=$((warnings + 1)); } +fail_check() { printf ' [FAIL] %s\n' "$1"; failures=$((failures + 1)); } + +check_command() { + local command_name="$1" remedy="$2" + if command -v "${command_name}" >/dev/null 2>&1; then + pass "${command_name} is installed" + else + fail_check "${command_name} is missing — ${remedy}" + fi +} + +check_directory() { + local path="$1" label="$2" remedy="$3" + if [[ -d "${path}" ]]; then pass "${label}: ${path}"; else fail_check "${label} is missing — ${remedy}"; fi +} + +case "${mode}" in + local|live) ;; + --help|-h) usage; exit 0 ;; + *) usage >&2; exit 2 ;; +esac + +echo "Sentinel judge demo doctor" +echo " Mode: ${mode}" +echo " Read-only: yes" +echo " Argus: ${repo_root}" +echo +echo "==> Repository layout" +check_directory "${repo_root}/agent" "Argus" "run this command from the argus-k8s checkout" +check_directory "${phoenix_root}/dashboard" "Phoenix" "clone Phoenix at ../sentinel-stack/phoenix or set PHOENIX_ROOT" +check_directory "${sentinel_root}/dashboard" "Sentinel" "clone Sentinel at ../sentinel-stack/sentinel or set SENTINEL_ROOT" +check_directory "${platform_root}/world_model" "Sentinel Platform / SOG" "clone Sentinel Platform at ../sentinel-stack/sentinel-platform or set SENTINEL_PLATFORM_ROOT" + +echo +echo "==> Required tools" +for tool in bash curl jq npm python3 lsof; do + check_command "${tool}" "install ${tool} and retry" +done + +if [[ -n "${OPENAI_API_KEY:-}" ]] || { [[ -f "${repo_root}/.env" ]] && grep -Eq '^[[:space:]]*OPENAI_API_KEY=.+' "${repo_root}/.env"; }; then + pass "OPENAI_API_KEY is configured (value not displayed)" +else + warn "OPENAI_API_KEY is not configured — deterministic proof works, but the OpenAI evidence briefing will be unavailable" +fi + +if [[ "${mode}" == "local" ]]; then + echo + echo "==> Portable demo runtime" + check_command docker "install Docker or start OrbStack" + if command -v docker >/dev/null 2>&1 && docker info >/dev/null 2>&1; then + pass "Docker-compatible runtime is reachable" + else + fail_check "Docker-compatible runtime is not reachable — start Docker or OrbStack" + fi +else + echo + echo "==> Local console dependencies" + [[ -x "${repo_root}/.venv/bin/python" ]] && pass "Argus Python environment is installed" || fail_check "Argus Python environment is missing — run: make setup-local" + [[ -d "${repo_root}/ui/node_modules" ]] && pass "Argus UI dependencies are installed" || fail_check "Argus UI dependencies are missing — run: make setup-local" + [[ -x "${sentinel_root}/.venv/bin/python" ]] && pass "Sentinel Python environment is installed" || fail_check "Sentinel Python environment is missing — run: make -C ${sentinel_root} setup-local" + [[ -d "${sentinel_root}/dashboard/node_modules" ]] && pass "Sentinel UI dependencies are installed" || fail_check "Sentinel UI dependencies are missing — run Sentinel setup-local" + [[ -d "${phoenix_root}/dashboard/node_modules" ]] && pass "Phoenix UI dependencies are installed" || fail_check "Phoenix UI dependencies are missing — run: npm --prefix ${phoenix_root}/dashboard install" + + echo + echo "==> Live k3s stack" + check_command kubectl "install kubectl and configure the argus context" + if command -v kubectl >/dev/null 2>&1; then + if LIVE_DEMO_DRY_RUN=true PHOENIX_ROOT="${phoenix_root}" SENTINEL_ROOT="${sentinel_root}" \ + bash "${repo_root}/scripts/demo-platform-live-proof.sh"; then + pass "live security, chaos, agent, and SOG preflight passed" + else + fail_check "live stack preflight failed — follow the component named above, then retry" + fi + fi +fi + +echo +echo "==> Verdict" +if ((failures > 0)); then + echo "NOT READY — ${failures} blocking check(s), ${warnings} warning(s)." + exit 1 +fi + +echo "READY — 0 blocking checks, ${warnings} warning(s)." +if [[ "${mode}" == "local" ]]; then + echo "Next: make demo-platform" +else + echo "Next: make demo-platform-live" + echo "The live command will still require the exact context and INJECT LIVE FAULT." +fi diff --git a/scripts/demo-platform-live-proof.sh b/scripts/demo-platform-live-proof.sh new file mode 100644 index 0000000..9d37a37 --- /dev/null +++ b/scripts/demo-platform-live-proof.sh @@ -0,0 +1,315 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +namespace="${LIVE_DEMO_NAMESPACE:-sentinel-live-demo}" +fault_seconds="${LIVE_FAULT_SECONDS:-15}" +evidence_wait_seconds="${LIVE_EVIDENCE_WAIT_SECONDS:-90}" +keep_resources="${LIVE_DEMO_KEEP_RESOURCES:-false}" +dry_run="${LIVE_DEMO_DRY_RUN:-false}" +authorized_context="${LIVE_DEMO_CONTEXT:-}" +authorized_phrase="${LIVE_DEMO_CONFIRMATION:-}" +log_dir="$(mktemp -d "${TMPDIR:-/tmp}/sentinel-live-proof.XXXXXX")" +platform_pid="" +probe_pid="" +target_forward_pid="" +namespace_created=false +scenario_id="" + +fail() { echo "ERROR: $*" >&2; exit 1; } +require_command() { command -v "$1" >/dev/null 2>&1 || fail "Missing command: $1"; } +iso_epoch() { python3 -c 'import datetime,sys; print(datetime.datetime.fromtimestamp(int(sys.argv[1]), datetime.timezone.utc).isoformat().replace("+00:00", "Z"))' "$1"; } + +cleanup() { + trap - INT TERM EXIT + for process_id in "${probe_pid}" "${target_forward_pid}" "${platform_pid}"; do + if [[ -n "${process_id}" ]]; then kill "${process_id}" 2>/dev/null || true; fi + done + for process_id in "${probe_pid}" "${target_forward_pid}" "${platform_pid}"; do + if [[ -n "${process_id}" ]]; then wait "${process_id}" 2>/dev/null || true; fi + done + if [[ "${namespace_created}" == "true" && "${keep_resources}" != "true" ]]; then + echo "==> Deleting isolated namespace ${namespace}" + kubectl --context "${context}" delete namespace "${namespace}" --ignore-not-found --wait=false >/dev/null || true + elif [[ "${namespace_created}" == "true" ]]; then + echo "Keeping ${namespace} because LIVE_DEMO_KEEP_RESOURCES=true." + fi + echo "Live proof stopped. Logs: ${log_dir}" +} +trap cleanup EXIT INT TERM + +case "${namespace}" in + default|kube-system|kube-public|kube-node-lease|argus-system|phoenix-system|sentinel-platform|kyverno|chaos-mesh) + fail "Refusing unsafe live-demo namespace: ${namespace}" ;; +esac +[[ "${namespace}" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]] || fail "LIVE_DEMO_NAMESPACE must be a DNS label." +[[ "${fault_seconds}" =~ ^[0-9]+$ ]] && ((fault_seconds >= 5 && fault_seconds <= 30)) || + fail "LIVE_FAULT_SECONDS must be between 5 and 30." +[[ "${evidence_wait_seconds}" =~ ^[0-9]+$ ]] && ((evidence_wait_seconds >= 30 && evidence_wait_seconds <= 180)) || + fail "LIVE_EVIDENCE_WAIT_SECONDS must be between 30 and 180." + +for required in kubectl curl jq awk npm python3; do require_command "${required}"; done +context="$(kubectl config current-context)" +server="$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')" + +echo "Sentinel guarded live resilience proof" +echo " Context: ${context}" +echo " API: ${server}" +echo " Namespace: ${namespace}" +echo " Fault: PodChaos / one of two demo replicas / ${fault_seconds}s" +echo " Cleanup: isolated namespace only" + +kubectl --context "${context}" --request-timeout=10s get --raw=/readyz >/dev/null +kubectl --context "${context}" wait --for=condition=Ready nodes --all --timeout=30s >/dev/null + +require_ready() { + local target_namespace="$1" selector="$2" description="$3" + kubectl --context "${context}" -n "${target_namespace}" wait --for=condition=Ready pod \ + -l "${selector}" --timeout=60s >/dev/null || fail "Required component is not ready: ${description}." + echo " ready: ${description}" +} + +echo "==> Validating real security, chaos, and correlation services" +require_ready kube-system k8s-app=cilium Cilium +require_ready kube-system app.kubernetes.io/name=falco Falco +require_ready kyverno app.kubernetes.io/component=admission-controller Kyverno +require_ready argus-system app=argus-agent "Argus agent" +require_ready kube-system app.kubernetes.io/component=controller-manager "Chaos Mesh controller" +require_ready kube-system app.kubernetes.io/component=chaos-daemon "Chaos Mesh daemon" +require_ready phoenix-system app=phoenix-chaos "Phoenix chaos engine" +require_ready phoenix-system app=phoenix-agent "Phoenix agent" +require_ready sentinel-platform app=sentinel-world-model "Sentinel Operations Graph" +kubectl --context "${context}" get crd podchaos.chaos-mesh.org >/dev/null +kubectl --context "${context}" auth can-i create podchaos.chaos-mesh.org -n "${namespace}" | grep -qx yes || + fail "Current identity cannot create PodChaos in ${namespace}." + +if [[ "${dry_run}" == "true" ]]; then + echo "Preflight passed. No namespace, workload, fault, process, or finding was created." + exit 0 +fi + +if [[ -n "${authorized_context}" ]]; then + [[ "${authorized_context}" == "${context}" ]] || fail "LIVE_DEMO_CONTEXT does not match ${context}." +elif [[ -t 0 ]]; then + read -r -p "Type the exact context name to authorize the isolated workload: " authorized_context + [[ "${authorized_context}" == "${context}" ]] || fail "Context confirmation failed." +else + fail "Non-interactive runs require LIVE_DEMO_CONTEXT=${context}." +fi + +kubectl --context "${context}" get namespace "${namespace}" >/dev/null 2>&1 && + fail "Namespace ${namespace} already exists; refusing to reuse or delete it." + +echo "==> Starting live cluster consoles without synthetic seed data" +DEMO_PLATFORM_CONTEXT="${context}" DEMO_PLATFORM_SKIP_SEED=true \ + bash "${repo_root}/scripts/demo-platform.sh" >"${log_dir}/platform.log" 2>&1 & +platform_pid=$! +for _attempt in $(seq 1 90); do + if curl --fail --silent --max-time 3 http://127.0.0.1:5175/api/health | + jq -e '.world_model_connected == true' >/dev/null 2>&1; then break; fi + kill -0 "${platform_pid}" 2>/dev/null || fail "Platform services exited. See ${log_dir}/platform.log." + sleep 1 +done +curl --fail --silent --max-time 3 http://127.0.0.1:5175/api/health | jq -e '.world_model_connected == true' >/dev/null || + fail "Sentinel UI did not connect through its API proxy." + +echo "==> Creating isolated two-replica customer service" +kubectl --context "${context}" create namespace "${namespace}" >/dev/null +namespace_created=true +kubectl --context "${context}" label namespace "${namespace}" sentinel.io/live-demo=true --overwrite >/dev/null +kubectl --context "${context}" apply -n "${namespace}" -f - >/dev/null <<'YAML' +apiVersion: apps/v1 +kind: Deployment +metadata: + name: sentinel-live-target +spec: + replicas: 2 + selector: + matchLabels: + app: sentinel-live-target + template: + metadata: + labels: + app: sentinel-live-target + sentinel.io/live-demo: "true" + spec: + securityContext: + runAsNonRoot: true + runAsUser: 101 + runAsGroup: 101 + fsGroup: 101 + containers: + - name: web + image: docker.io/nginxinc/nginx-unprivileged:1.27-alpine + ports: + - containerPort: 8080 + readinessProbe: + httpGet: + path: / + port: 8080 + periodSeconds: 1 + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: ["ALL"] + readOnlyRootFilesystem: false + resources: + requests: {cpu: 25m, memory: 32Mi} + limits: {cpu: 100m, memory: 96Mi} +--- +apiVersion: v1 +kind: Service +metadata: + name: sentinel-live-target +spec: + selector: + app: sentinel-live-target + ports: + - port: 8080 + targetPort: 8080 +YAML +kubectl --context "${context}" -n "${namespace}" rollout status deployment/sentinel-live-target --timeout=120s >/dev/null + +kubectl --context "${context}" -n "${namespace}" port-forward --address 127.0.0.1 \ + svc/sentinel-live-target 18080:8080 >"${log_dir}/target-forward.log" 2>&1 & +target_forward_pid=$! +for _attempt in $(seq 1 30); do + curl --fail --silent --max-time 2 http://127.0.0.1:18080/ >/dev/null && break + sleep 1 +done +curl --fail --silent --max-time 2 http://127.0.0.1:18080/ >/dev/null || fail "Demo service is not reachable." +healthy_epoch="$(date +%s)" + +probe_log="${log_dir}/availability.log" +( + while true; do + if curl --fail --silent --max-time 1 http://127.0.0.1:18080/ >/dev/null; then echo 1; else echo 0; fi + sleep 0.25 + done +) >"${probe_log}" & +probe_pid=$! + +echo "==> Launching bounded real Falco-triggering workload" +fault_injected_epoch="$(date +%s)" +DEMO_NAMESPACE="${namespace}" bash "${repo_root}/cluster/test-diverse-threats.sh" >"${log_dir}/threats.log" 2>&1 +kubectl --context "${context}" -n "${namespace}" wait --for=condition=Ready pod -l threat-type --timeout=120s >/dev/null + +argus_incident_file="${log_dir}/argus-incident.json" +argus_detected=false +evidence_started_epoch="${fault_injected_epoch}" +for _attempt in $(seq 1 "${evidence_wait_seconds}"); do + if curl --fail --silent --max-time 10 http://127.0.0.1:8000/incidents | + jq -e --arg namespace "${namespace}" '[.[] | select((.namespace // "") == $namespace)] | first' >"${argus_incident_file}"; then + argus_detected=true + break + fi + sleep 1 +done +[[ "${argus_detected}" == "true" ]] || fail "Argus did not expose Falco evidence for ${namespace} within ${evidence_wait_seconds}s." +argus_detected_epoch="$(date +%s)" +decision_epoch="$(date +%s)" + +if [[ -z "${authorized_phrase}" && -t 0 ]]; then + echo "Observed Argus evidence is now present. The next approval permits one real PodChaos against one of two disposable replicas." + read -r -p "Type INJECT LIVE FAULT to authorize recovery: " authorized_phrase +fi +[[ "${authorized_phrase}" == "INJECT LIVE FAULT" ]] || fail "Live fault confirmation failed." +approved_epoch="$(date +%s)" + +old_pods="$(kubectl --context "${context}" -n "${namespace}" get pods -l app=sentinel-live-target -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')" +correlation_id="live-proof-$(date -u +%Y%m%dT%H%M%SZ)-$$" +echo "==> Injecting one real, approved PodChaos" +scenario="$(curl --fail --silent --show-error -X POST http://127.0.0.1:8082/scenarios \ + -H 'Content-Type: application/json' \ + -d "$(jq -nc --arg corr "${correlation_id}" --arg namespace "${namespace}" --argjson duration "${fault_seconds}" '{name:"Sentinel live pod recovery proof",correlation_id:$corr,domain:"chaos_mesh",fault_type:"pod_kill",target:{namespace:$namespace,label_selector:{app:"sentinel-live-target"},mode:"one"},duration_seconds:$duration,params:{grace_period_seconds:0}}')")" +scenario_id="$(jq -r '.id' <<<"${scenario}")" +backend_ref="$(jq -r '.backend_ref' <<<"${scenario}")" +fault_started_epoch="$(date +%s)" +[[ -n "${scenario_id}" && "${scenario_id}" != "null" && -n "${backend_ref}" && "${backend_ref}" != "null" ]] || + fail "Phoenix did not return a scenario ID and Chaos Mesh backend reference." +kubectl --context "${context}" -n "${namespace}" get podchaos "${backend_ref}" -o json >"${log_dir}/podchaos.json" || + fail "Phoenix returned ${backend_ref}, but the real PodChaos object was not found." + +replacement_ready=false +replacement_name="" +for _attempt in $(seq 1 120); do + replacement_name="$(kubectl --context "${context}" -n "${namespace}" get pods -l app=sentinel-live-target \ + --field-selector=status.phase=Running -o jsonpath='{range .items[?(@.status.containerStatuses[0].ready==true)]}{.metadata.name}{"\n"}{end}' 2>/dev/null | + while IFS= read -r pod_name; do grep -qxF "${pod_name}" <<<"${old_pods}" || { echo "${pod_name}"; break; }; done)" + ready_replicas="$(kubectl --context "${context}" -n "${namespace}" get deployment sentinel-live-target -o jsonpath='{.status.readyReplicas}' 2>/dev/null || true)" + if [[ -n "${replacement_name}" && "${ready_replicas:-0}" == "2" ]]; then replacement_ready=true; break; fi + sleep 1 +done +[[ "${replacement_ready}" == "true" ]] || fail "A replacement replica was not verified after PodChaos." +recovery_epoch="$(date +%s)" +recovery_seconds="$((recovery_epoch - fault_started_epoch))" +sleep 5 + +probe_total="$(wc -l <"${probe_log}" | tr -d ' ')" +probe_success="$(awk '$1 == 1 {count++} END {print count+0}' "${probe_log}")" +((probe_total > 0)) || fail "Availability probe produced no samples." +availability="$(awk -v success="${probe_success}" -v total="${probe_total}" 'BEGIN {printf "%.2f", (success/total)*100}')" +verification_epoch="$(date +%s)" + +echo "==> Correlating verified live evidence through Sentinel" +entity_id="service/${namespace}/sentinel-live-target" +timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +curl --fail --silent -X POST http://127.0.0.1:8010/entities -H 'Content-Type: application/json' \ + -d "$(jq -nc --arg id "${entity_id}" --arg namespace "${namespace}" '{entity_id:$id,entity_type:"service",name:"sentinel-live-target",namespace:$namespace,labels:{"sentinel.io/live-demo":"true"},slo_target:0.999}')" >/dev/null +curl --fail --silent -X POST http://127.0.0.1:8010/findings -H 'Content-Type: application/json' \ + -d "$(jq -nc --arg id "argus-${correlation_id}" --arg ts "${timestamp}" --arg entity "${entity_id}" --arg corr "${correlation_id}" --slurpfile original "${argus_incident_file}" '{event_id:$id,type:"finding",source:"argus",timestamp:$ts,entity_id:$entity,severity:"high",correlation_id:$corr,replayed:false,payload:{finding_type:"falco_alert",assessment:"Observed Falco runtime evidence in isolated live-demo namespace",provenance:"observed",evidence_source:"argus_incidents_api",original_incident:$original[0]}}')" >/dev/null +curl --fail --silent -X POST http://127.0.0.1:8010/findings -H 'Content-Type: application/json' \ + -d "$(jq -nc --arg id "phoenix-${correlation_id}" --arg ts "${timestamp}" --arg entity "${entity_id}" --arg corr "${correlation_id}" --arg scenario "${scenario_id}" --arg backend "${backend_ref}" --arg replacement "${replacement_name}" --arg availability "${availability}" --arg healthy "$(iso_epoch "${healthy_epoch}")" --arg fault "$(iso_epoch "${fault_injected_epoch}")" --arg detected "$(iso_epoch "${argus_detected_epoch}")" --arg decision "$(iso_epoch "${decision_epoch}")" --arg approval "$(iso_epoch "${approved_epoch}")" --arg recovered "$(iso_epoch "${recovery_epoch}")" --arg verified "$(iso_epoch "${verification_epoch}")" --argjson detection_ms "$(((argus_detected_epoch - fault_injected_epoch) * 1000))" --argjson recovery_seconds "${recovery_seconds}" '{event_id:$id,type:"finding",source:"phoenix",timestamp:$ts,entity_id:$entity,severity:"high",correlation_id:$corr,payload:{finding_type:"healing_action",outcome:"verified_recovery",description:"Chaos Mesh killed one disposable replica; Kubernetes replaced it and service health was verified",provenance:"live_chaos",domain:"chaos_mesh",scenario_id:$scenario,backend_ref:$backend,replacement_pod:$replacement,recovery_seconds:$recovery_seconds,measured_availability_percent:$availability,approval_required:true,approval_record:"INJECT LIVE FAULT",evidence_source:"Falco + Argus + Phoenix + Kubernetes + HTTP probe",metrics:{detection_ms:$detection_ms,recovery_ms:($recovery_seconds*1000),availability_percent:($availability|tonumber)},lifecycle:[{stage:"healthy",timestamp:$healthy,source:"HTTP probe",evidence:"Two Ready replicas served successful HTTP probes",status:"verified"},{stage:"fault_injected",timestamp:$fault,source:"bounded workload",evidence:"Isolated Falco-triggering workload started",status:"observed"},{stage:"detection",timestamp:$detected,source:"Argus + Falco",evidence:"Argus incidents API exposed observed runtime evidence",status:"observed"},{stage:"decision",timestamp:$decision,source:"Sentinel guardrail",evidence:"Bounded one-of-two recovery action required human approval",status:"gated"},{stage:"human_approval",timestamp:$approval,source:"operator",evidence:"Operator typed INJECT LIVE FAULT after detection",status:"approved"},{stage:"recovery",timestamp:$recovered,source:"Phoenix + Kubernetes",evidence:("Replacement pod " + $replacement + " became Ready"),status:"recovered"},{stage:"verification",timestamp:$verified,source:"HTTP probe + Kubernetes",evidence:"Two replicas Ready and continuous availability calculated",status:"verified"}]}}')" >/dev/null + +incident_file="${log_dir}/sentinel-incident.json" +verified=false +for _attempt in $(seq 1 60); do + if curl --fail --silent --max-time 25 http://127.0.0.1:5175/api/overview | + jq -e --arg correlation_id "${correlation_id}" '.incidents[] | select(.correlation_id == $correlation_id and .sources == ["argus","phoenix"] and .provenance == ["live_chaos","observed"] and (.proof.lifecycle | length) == 7 and (.proof.metrics.availability_percent | type) == "number")' >"${incident_file}"; then + verified=true; break + fi + sleep 1 +done +[[ "${verified}" == "true" ]] || fail "Sentinel did not expose live correlation ${correlation_id}." +sentinel_health="$(curl --fail --silent --max-time 10 http://127.0.0.1:5175/api/health)" +openai_configured="$(jq -r '.openai_configured == true' <<<"${sentinel_health}")" + +artifact_dir="${repo_root}/artifacts/demo-platform" +mkdir -p "${artifact_dir}" +report_json="${artifact_dir}/${correlation_id}.json" +report_markdown="${artifact_dir}/${correlation_id}.md" +jq -n --arg run_id "${correlation_id}" --arg generated_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg correlation_id "${correlation_id}" --arg entity_id "${entity_id}" --arg context "${context}" \ + --arg namespace "${namespace}" --arg availability "${availability}" --arg scenario "${scenario_id}" \ + --argjson openai_configured "${openai_configured}" \ + --argjson detection_ms "$(((argus_detected_epoch - evidence_started_epoch) * 1000))" \ + --argjson recovery_ms "$((recovery_seconds * 1000))" --slurpfile incident "${incident_file}" \ + '{schema_version:"1.0",verdict:"PASS",run_id:$run_id,generated_at:$generated_at, + cluster:{context:$context,namespace:$namespace,nodes:3},correlation:{id:$correlation_id,verified:true,sources:["argus","phoenix"],incident:$incident[0]}, + entity:{id:$entity_id},timings:{argus_detection_ms:$detection_ms,recovery_ms:$recovery_ms,sentinel_correlation_ms:0,total_lifecycle_ms:($detection_ms+$recovery_ms)}, + recovery:{result:"verified_recovery",availability:("measured " + $availability + "%")}, + governance:{approval_required:true,reason:"operator typed INJECT LIVE FAULT"},openai:{configured:$openai_configured,briefing_generated:false}, + evidence:{provenance:["live_chaos","observed"],seed:null,live_chaos:true,scenario_id:$scenario}, + interpretation:"Real Falco evidence and a real Chaos Mesh PodChaos were executed only inside an isolated namespace. Availability came from continuous HTTP probes; recovery required a new Ready replica and two Ready deployment replicas."}' >"${report_json}" +bash "${repo_root}/scripts/render-demo-report.sh" "${report_json}" "${report_markdown}" +cp "${report_json}" "${artifact_dir}/latest-live-demo.json" +cp "${report_markdown}" "${artifact_dir}/latest-live-demo.md" + +echo "" +echo "LIVE PLATFORM RESILIENCE PROOF: PASS" +echo " Cluster: ${context} — 3 nodes" +echo " Namespace: ${namespace}" +echo " Argus detection: observed via Falco" +echo " Phoenix experiment: live_chaos / ${scenario_id}" +echo " Service availability: ${availability}% (${probe_success}/${probe_total} probes)" +echo " Recovery verified: ${replacement_name} ready" +echo " Recovery time: ${recovery_seconds}s" +echo " Human approval: explicitly authorized" +echo " Sentinel correlation: ${correlation_id}" +echo " Evidence: ${artifact_dir}/latest-live-demo.md" +echo " Cleanup boundary: namespace/${namespace}" +echo "" +echo "Open Sentinel at http://127.0.0.1:5175 and select the live correlated incident." +echo "Press Ctrl-C to stop consoles and delete only the isolated namespace." + +while true; do sleep 60; done diff --git a/scripts/demo-platform-local.sh b/scripts/demo-platform-local.sh new file mode 100755 index 0000000..2ee7d63 --- /dev/null +++ b/scripts/demo-platform-local.sh @@ -0,0 +1,258 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +argus_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +phoenix_root="${PHOENIX_ROOT:-${argus_root}/../sentinel-stack/phoenix}" +sentinel_root="${SENTINEL_ROOT:-${argus_root}/../sentinel-stack/sentinel}" +platform_root="${SENTINEL_PLATFORM_ROOT:-${argus_root}/../sentinel-stack/sentinel-platform}" +dry_run="${DEMO_PLATFORM_DRY_RUN:-false}" +exit_after_ready="${DEMO_PLATFORM_EXIT_AFTER_READY:-false}" +log_dir="$(mktemp -d "${TMPDIR:-/tmp}/sentinel-local-demo.XXXXXX")" +artifact_dir="${DEMO_PLATFORM_ARTIFACT_DIR:-${argus_root}/artifacts/demo-platform}" +redis_port="${DEMO_REDIS_PORT:-6389}" +redis_name="sentinel-local-demo-$$" +owned_pids=() +redis_owned=false + +cleanup() { + trap - INT TERM EXIT + if ((${#owned_pids[@]})); then + echo "" + echo "Stopping command-owned local demo processes ..." + for owned_pid in "${owned_pids[@]}"; do kill "${owned_pid}" 2>/dev/null || true; done + for owned_pid in "${owned_pids[@]}"; do wait "${owned_pid}" 2>/dev/null || true; done + fi + if [[ "${redis_owned}" == "true" ]]; then + docker stop --time 2 "${redis_name}" >/dev/null 2>&1 || true + fi + echo "Local platform demo stopped. Logs: ${log_dir}" +} +trap cleanup EXIT INT TERM + +fail() { echo "ERROR: $*" >&2; exit 1; } +require_command() { command -v "$1" >/dev/null 2>&1 || fail "Missing command: $1"; } +http_ok() { curl --fail --silent --max-time 3 "$1" >/dev/null 2>&1; } +port_in_use() { lsof -nP -iTCP:"$1" -sTCP:LISTEN >/dev/null 2>&1; } +now_ms() { python3 -c 'import time; print(time.time_ns() // 1_000_000)'; } + +wait_for_url() { + local label="$1" url="$2" pid="${3:-}" attempt + for attempt in $(seq 1 60); do + if http_ok "${url}"; then return 0; fi + if [[ -n "${pid}" ]] && ! kill -0 "${pid}" 2>/dev/null; then + fail "${label} exited before becoming ready. See ${log_dir}." + fi + sleep 1 + done + fail "${label} did not become ready at ${url}. See ${log_dir}." +} + +stop_known_listener() { + local port="$1" expected_cwd="$2" expected_command="$3" listener_pid listener_cwd listener_command + port_in_use "${port}" || return 0 + listener_pid="$(lsof -nP -tiTCP:"${port}" -sTCP:LISTEN | head -1)" + listener_cwd="$(lsof -a -p "${listener_pid}" -d cwd -Fn 2>/dev/null | sed -n 's/^n//p')" + listener_command="$(ps -p "${listener_pid}" -o command= 2>/dev/null || true)" + [[ "${listener_cwd}" == "${expected_cwd}" && "${listener_command}" == *"${expected_command}"* ]] || + fail "Port ${port} belongs to PID ${listener_pid}, not the expected ${expected_command} process in ${expected_cwd}." + echo " replace: ${expected_command} listener on 127.0.0.1:${port}" + kill "${listener_pid}" + for _attempt in $(seq 1 20); do port_in_use "${port}" || return 0; sleep 0.25; done + fail "PID ${listener_pid} did not release port ${port}." +} + +stop_cluster_forward() { + local port="$1" listener_pid listener_command + port_in_use "${port}" || return 0 + listener_pid="$(lsof -nP -tiTCP:"${port}" -sTCP:LISTEN | head -1)" + listener_command="$(ps -p "${listener_pid}" -o command= 2>/dev/null || true)" + [[ "${listener_command}" == *"kubectl"*"port-forward"* ]] || + fail "Port ${port} is occupied by PID ${listener_pid}; local mode will not terminate an unrelated listener." + echo " replace: Kubernetes port-forward on 127.0.0.1:${port}" + kill "${listener_pid}" + for _attempt in $(seq 1 20); do port_in_use "${port}" || return 0; sleep 0.25; done + fail "Port-forward PID ${listener_pid} did not release port ${port}." +} + +ensure_venv() { + local service_root="$1" requirements="$2" + if [[ ! -x "${service_root}/.venv/bin/python" ]]; then + echo " install: Python dependencies in ${service_root}" + python3 -m venv "${service_root}/.venv" + "${service_root}/.venv/bin/pip" install -r "${requirements}" + elif ! "${service_root}/.venv/bin/python" -c 'import fastapi, uvicorn' >/dev/null 2>&1; then + echo " repair: incomplete Python dependencies in ${service_root}" + "${service_root}/.venv/bin/pip" install -r "${requirements}" + fi +} + +ensure_node_modules() { + local package_root="$1" + if [[ ! -d "${package_root}/node_modules" ]]; then + echo " install: npm dependencies in ${package_root}" + npm --prefix "${package_root}" install + fi +} + +start_service() { + local label="$1" port="$2" health_url="$3" workdir="$4" + shift 4 + port_in_use "${port}" && fail "Port ${port} is still occupied before starting ${label}." + echo " start: ${label} on 127.0.0.1:${port}" + (cd "${workdir}" && exec "$@") >"${log_dir}/${label}.log" 2>&1 & + local service_pid=$! + owned_pids+=("${service_pid}") + wait_for_url "${label}" "${health_url}" "${service_pid}" +} + +echo "Sentinel platform local judge demo" +echo " Mode: cluster-free synthetic topology + deterministic evidence" +echo " Kubernetes: not used" +echo " Argus: ${argus_root}" +echo " Phoenix: ${phoenix_root}" +echo " Sentinel: ${sentinel_root}" +echo " SOG: ${platform_root}" + +for required in curl jq lsof npm python3 docker ps; do require_command "${required}"; done +[[ -d "${phoenix_root}/dashboard" ]] || fail "Phoenix checkout not found. Set PHOENIX_ROOT." +[[ -d "${sentinel_root}/dashboard" ]] || fail "Sentinel checkout not found. Set SENTINEL_ROOT." +[[ -d "${platform_root}/world_model" ]] || fail "Sentinel Platform checkout not found. Set SENTINEL_PLATFORM_ROOT." +docker info >/dev/null 2>&1 || fail "Docker is not running. Start Docker or OrbStack, then retry." + +if [[ "${dry_run}" == "true" ]]; then + echo "Preflight passed. No cluster, containers, processes, or evidence were changed." + exit 0 +fi + +if [[ -f "${argus_root}/.env" ]]; then set -a; source "${argus_root}/.env"; set +a; fi + +echo "==> Installing missing local dependencies" +ensure_venv "${argus_root}" "${argus_root}/agent/requirements.txt" +ensure_venv "${platform_root}" "${platform_root}/world_model/requirements.txt" +ensure_venv "${sentinel_root}" "${sentinel_root}/backend/requirements.txt" +for phoenix_service in graph sim chaos faultlib agent; do + ensure_venv "${phoenix_root}/${phoenix_service}" "${phoenix_root}/${phoenix_service}/requirements.txt" +done +ensure_node_modules "${argus_root}/ui" +ensure_node_modules "${phoenix_root}/dashboard" +ensure_node_modules "${sentinel_root}/dashboard" + +echo "==> Replacing only known platform listeners" +stop_cluster_forward 8010 +for service_port in 8080 8081 8082 8083 8084; do stop_cluster_forward "${service_port}"; done +stop_known_listener 8000 "${argus_root}/agent/src" "uvicorn" +stop_known_listener 5173 "${argus_root}/ui" "vite" +stop_known_listener 5174 "${phoenix_root}/dashboard" "vite" +stop_known_listener 8090 "${sentinel_root}" "uvicorn" +stop_known_listener 5175 "${sentinel_root}/dashboard" "vite" +port_in_use "${redis_port}" && fail "Redis demo port ${redis_port} is already occupied." + +echo "==> Starting disposable local SOG" +docker run --rm --name "${redis_name}" -p "127.0.0.1:${redis_port}:6379" redis:7-alpine \ + >"${log_dir}/redis.log" 2>&1 & +redis_pid=$! +owned_pids+=("${redis_pid}") +redis_owned=true +for _attempt in $(seq 1 40); do + docker exec "${redis_name}" redis-cli ping 2>/dev/null | grep -q PONG && break + kill -0 "${redis_pid}" 2>/dev/null || fail "Disposable Redis exited. See ${log_dir}/redis.log." + sleep 0.5 +done +docker exec "${redis_name}" redis-cli ping 2>/dev/null | grep -q PONG || fail "Disposable Redis did not become ready." + +start_service sog 8010 http://127.0.0.1:8010/health "${platform_root}/world_model/src" \ + env WM_REDIS_URL="redis://127.0.0.1:${redis_port}" "${platform_root}/.venv/bin/python" -m uvicorn main:app --host 127.0.0.1 --port 8010 + +echo "==> Starting local Phoenix services" +start_service phoenix-graph 8080 http://127.0.0.1:8080/health "${phoenix_root}/graph/src" \ + env PHOENIX_LOCAL_DEMO=true "${phoenix_root}/graph/.venv/bin/python" -m uvicorn main:app --host 127.0.0.1 --port 8080 +start_service phoenix-sim 8083 http://127.0.0.1:8083/health "${phoenix_root}/sim/src" \ + "${phoenix_root}/sim/.venv/bin/python" -m uvicorn main:app --host 127.0.0.1 --port 8083 +start_service phoenix-chaos 8082 http://127.0.0.1:8082/health "${phoenix_root}/chaos/src" \ + env SIMULATOR_URL=http://127.0.0.1:8083 "${phoenix_root}/chaos/.venv/bin/python" -m uvicorn main:app --host 127.0.0.1 --port 8082 +start_service phoenix-faultlib 8081 http://127.0.0.1:8081/health "${phoenix_root}/faultlib/src" \ + env CHAOS_URL=http://127.0.0.1:8082 "${phoenix_root}/faultlib/.venv/bin/python" -m uvicorn main:app --host 127.0.0.1 --port 8081 +start_service phoenix-agent 8084 http://127.0.0.1:8084/health "${phoenix_root}" \ + env CHAOS_URL=http://127.0.0.1:8082 GRAPH_URL=http://127.0.0.1:8080 FAULTLIB_URL=http://127.0.0.1:8081 \ + WORLD_MODEL_URL=http://127.0.0.1:8010 DB_PATH="${log_dir}/phoenix-memory.db" OPENAI_API_KEY="${OPENAI_API_KEY:-}" \ + "${phoenix_root}/agent/.venv/bin/python" -m uvicorn main:app --app-dir agent/src --host 127.0.0.1 --port 8084 + +echo "==> Starting specialist and command-center consoles" +start_service argus-api 8000 http://127.0.0.1:8000/health "${argus_root}/agent/src" \ + env ARGUS_LOCAL_DEMO=true IN_CLUSTER=false WORLD_MODEL_URL=http://127.0.0.1:8010 \ + "${argus_root}/.venv/bin/python" -m uvicorn main:app --host 127.0.0.1 --port 8000 +start_service argus-ui 5173 http://127.0.0.1:5173/api/health "${argus_root}" \ + npm --prefix ui run dev -- --host 127.0.0.1 --port 5173 +start_service phoenix-ui 5174 http://127.0.0.1:5174/api/graph/health "${phoenix_root}" \ + env VITE_ARGUS_URL=http://127.0.0.1:5173 VITE_SENTINEL_URL=http://127.0.0.1:5175 \ + npm --prefix dashboard run dev -- --host 127.0.0.1 --port 5174 +start_service sentinel-api 8090 http://127.0.0.1:8090/health "${sentinel_root}" \ + env WORLD_MODEL_URL=http://127.0.0.1:8010 OPENAI_API_KEY="${OPENAI_API_KEY:-}" \ + "${sentinel_root}/.venv/bin/python" -m uvicorn main:app --app-dir backend/src --host 127.0.0.1 --port 8090 +start_service sentinel-ui 5175 http://127.0.0.1:5175/api/health "${sentinel_root}" \ + env VITE_ARGUS_URL=http://127.0.0.1:5173 VITE_PHOENIX_URL=http://127.0.0.1:5174 \ + npm --prefix dashboard run dev -- --host 127.0.0.1 --port 5175 + +echo "==> Seeding the synthetic cluster and cross-agent lifecycles" +demo_started_ms="$(now_ms)" +run_id="$(DEMO_SEED=42 bash "${argus_root}/scripts/seed-platform-local.sh")" +seeded_ms="$(now_ms)" +correlation_id="local-${run_id}-checkout-egress" +verified=false +incident_file="${log_dir}/verified-incident.json" +for _attempt in $(seq 1 60); do + if curl --fail --silent --max-time 25 http://127.0.0.1:5175/api/overview | + jq -e --arg correlation_id "${correlation_id}" '.incidents[] | select(.correlation_id == $correlation_id and .sources == ["argus","phoenix"] and .provenance == ["replayed","simulator"] and (.proof.lifecycle | length) == 7 and .proof.metrics.availability_percent == "not measured")' >"${incident_file}"; then + verified=true + break + fi + sleep 1 +done +[[ "${verified}" == "true" ]] || fail "Sentinel UI did not expose local correlation ${correlation_id}." +verified_ms="$(now_ms)" + +mkdir -p "${artifact_dir}" +run_json="${artifact_dir}/${correlation_id}.json" +run_markdown="${artifact_dir}/${correlation_id}.md" +openai_configured="$(curl --fail --silent http://127.0.0.1:8090/health | jq -r '.openai_configured == true')" +jq -n --arg run_id "${run_id}" --arg generated_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg correlation_id "${correlation_id}" --argjson openai_configured "${openai_configured}" \ + --argjson seed_ms "$((seeded_ms - demo_started_ms))" --argjson correlation_ms "$((verified_ms - seeded_ms))" \ + --argjson lifecycle_ms "$((verified_ms - demo_started_ms))" --slurpfile incident "${incident_file}" \ + '{schema_version:"1.0",verdict:"PASS",run_id:$run_id,generated_at:$generated_at, + correlation:{id:$correlation_id,verified:true,sources:["argus","phoenix"],incident:$incident[0]}, + entity:{id:"service/storefront/checkout"}, + timings:{argus_evidence_publish_ms:$seed_ms,phoenix_recovery_publish_ms:0,sentinel_correlation_ms:$correlation_ms,total_lifecycle_ms:$lifecycle_ms}, + recovery:{result:"verified_recovery",availability:"recovery verified; availability percentage not measured"}, + governance:{approval_required:false,reason:"bounded simulator action"},openai:{configured:$openai_configured,briefing_generated:false}, + evidence:{provenance:["replayed","simulator"],seed:42,live_chaos:false}, + interpretation:"A real local SOG and all three product services processed an explicitly synthetic cluster topology, replayed Argus evidence, and Phoenix simulator outcomes. No Kubernetes API or live fault was used."}' >"${run_json}" +bash "${argus_root}/scripts/render-demo-report.sh" "${run_json}" "${run_markdown}" +cp "${run_json}" "${artifact_dir}/latest-demo.json" +cp "${run_markdown}" "${artifact_dir}/latest-demo.md" + +DEMO_SEED=42 bash "${argus_root}/scripts/platform-local-feed.sh" >"${log_dir}/feed.log" 2>&1 & +feed_pid=$! +owned_pids+=("${feed_pid}") + +overview="$(curl --fail --silent --max-time 25 http://127.0.0.1:5175/api/overview)" +echo "" +echo "PLATFORM LOCAL PROOF: PASS" +echo " Kubernetes API calls: 0" +printf ' Synthetic topology: %s entities · %s relationships · %s namespaces\n' \ + "$(jq -r '.counts.entities' <<<"${overview}")" "$(jq -r '.counts.edges' <<<"${overview}")" "$(jq -r '.counts.namespaces' <<<"${overview}")" +printf ' Seeded evidence: %s findings · %s incidents\n' \ + "$(jq -r '.counts.findings' <<<"${overview}")" "$(jq -r '.counts.incidents' <<<"${overview}")" +echo " Recovery: verified in simulator" +echo " Provenance: replayed + simulator (seed 42)" +echo " Dynamic feed: every ${DEMO_FEED_INTERVAL_SECONDS:-20}s" +echo " Evidence: ${artifact_dir}/latest-demo.md" +echo "" +echo "Open Argus → http://127.0.0.1:5173" +echo "Open Phoenix → http://127.0.0.1:5174" +echo "Open Sentinel → http://127.0.0.1:5175" +echo "Synthetic data is labeled in every topology node and evidence payload." +echo "Press Ctrl-C to stop the entire disposable local platform." + +if [[ "${exit_after_ready}" == "true" ]]; then exit 0; fi +while true; do sleep 60; done diff --git a/scripts/demo-platform.sh b/scripts/demo-platform.sh index 9814a82..ae92ebe 100755 --- a/scripts/demo-platform.sh +++ b/scripts/demo-platform.sh @@ -6,8 +6,10 @@ phoenix_root="${PHOENIX_ROOT:-$(cd "${argus_root}/../sentinel-stack/phoenix" 2>/ sentinel_root="${SENTINEL_ROOT:-$(cd "${argus_root}/../sentinel-stack/sentinel" 2>/dev/null && pwd)}" dry_run="${DEMO_PLATFORM_DRY_RUN:-false}" exit_after_ready="${DEMO_PLATFORM_EXIT_AFTER_READY:-false}" +skip_seed="${DEMO_PLATFORM_SKIP_SEED:-false}" demo_context="${DEMO_PLATFORM_CONTEXT:-$(kubectl config current-context 2>/dev/null || true)}" log_dir="$(mktemp -d "${TMPDIR:-/tmp}/sentinel-demo.XXXXXX")" +artifact_dir="${DEMO_PLATFORM_ARTIFACT_DIR:-${argus_root}/artifacts/demo-platform}" owned_pids=() cleanup() { @@ -30,6 +32,31 @@ fail() { echo "ERROR: $*" >&2; exit 1; } require_command() { command -v "$1" >/dev/null 2>&1 || fail "Missing command: $1"; } http_ok() { curl --fail --silent --max-time 2 "$1" >/dev/null 2>&1; } port_in_use() { lsof -nP -iTCP:"$1" -sTCP:LISTEN >/dev/null 2>&1; } +now_ms() { python3 -c 'import time; print(time.time_ns() // 1_000_000)'; } +sentinel_ui_ok() { + curl --fail --silent --max-time 3 http://127.0.0.1:5175/api/health | + jq -e '.service == "sentinel-orchestrator" and (.world_model_connected | type == "boolean")' >/dev/null 2>&1 +} + +reclaim_stale_sentinel_ui() { + local listener_pid listener_cwd expected_cwd="${sentinel_root}/dashboard" + sentinel_ui_ok && return 0 + port_in_use 5175 || return 0 + + listener_pid="$(lsof -nP -tiTCP:5175 -sTCP:LISTEN | head -1)" + [[ -n "${listener_pid}" ]] || fail "Port 5175 is occupied but its listener cannot be identified." + listener_cwd="$(lsof -a -p "${listener_pid}" -d cwd -Fn 2>/dev/null | sed -n 's/^n//p')" + [[ "${listener_cwd}" == "${expected_cwd}" ]] || + fail "Port 5175 is occupied by PID ${listener_pid} outside ${expected_cwd}; stop it manually." + + echo " restart: stale Sentinel UI on 127.0.0.1:5175 (API proxy is not serving JSON)" + kill "${listener_pid}" + for _attempt in $(seq 1 20); do + port_in_use 5175 || return 0 + sleep 0.25 + done + fail "Stale Sentinel UI PID ${listener_pid} did not release port 5175." +} wait_for_url() { local label="$1" url="$2" pid="${3:-}" attempt @@ -128,34 +155,99 @@ start_service "phoenix-ui" 5174 http://127.0.0.1:5174 "${phoenix_root}" \ start_service "sentinel-api" 8090 http://127.0.0.1:8090/health "${sentinel_root}" \ env WORLD_MODEL_URL=http://127.0.0.1:8010 OPENAI_API_KEY="${OPENAI_API_KEY:-}" \ "${sentinel_root}/.venv/bin/python" -m uvicorn main:app --app-dir backend/src --host 127.0.0.1 --port 8090 -start_service "sentinel-ui" 5175 http://127.0.0.1:5175 "${sentinel_root}" \ +reclaim_stale_sentinel_ui +start_service "sentinel-ui" 5175 http://127.0.0.1:5175/api/health "${sentinel_root}" \ env VITE_ARGUS_URL=http://127.0.0.1:5173 VITE_PHOENIX_URL=http://127.0.0.1:5174 \ npm --prefix dashboard run dev -- --host 127.0.0.1 --port 5175 +sentinel_ui_ok || fail "Sentinel UI proxy did not return orchestrator JSON from /api/health." + +if [[ "${skip_seed}" == "true" ]]; then + echo "Live platform services are ready; waiting for the guarded experiment driver." + if [[ "${exit_after_ready}" == "true" ]]; then exit 0; fi + while true; do sleep 60; done +fi run_id="$(date -u +%Y%m%dT%H%M%SZ)-$$" correlation_id="judge-demo-${run_id}" timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)" entity_id="service/phoenix-system/phoenix-sim" +demo_started_ms="$(now_ms)" echo "==> Publishing deterministic correlated evidence" curl --fail --silent --show-error -X POST http://127.0.0.1:8010/findings \ -H "Content-Type: application/json" \ -d "{\"event_id\":\"argus-${run_id}\",\"type\":\"finding\",\"source\":\"argus\",\"timestamp\":\"${timestamp}\",\"entity_id\":\"${entity_id}\",\"severity\":\"critical\",\"correlation_id\":\"${correlation_id}\",\"replayed\":true,\"payload\":{\"finding_type\":\"falco_alert\",\"rule\":\"Deterministic C2 Callback Proof\",\"description\":\"Replayed Argus evidence for the judge demo\",\"provenance\":\"replayed\",\"seed\":42}}" >/dev/null +argus_published_ms="$(now_ms)" curl --fail --silent --show-error -X POST http://127.0.0.1:8010/findings \ -H "Content-Type: application/json" \ - -d "{\"event_id\":\"phoenix-${run_id}\",\"type\":\"finding\",\"source\":\"phoenix\",\"timestamp\":\"${timestamp}\",\"entity_id\":\"${entity_id}\",\"severity\":\"high\",\"correlation_id\":\"${correlation_id}\",\"payload\":{\"finding_type\":\"healing_action\",\"scenario_id\":\"sim-${run_id}\",\"outcome\":\"verified_recovery\",\"description\":\"Phoenix simulator verified service recovery\",\"provenance\":\"simulator\",\"domain\":\"simulator\",\"seed\":42}}" >/dev/null + -d "{\"event_id\":\"phoenix-${run_id}\",\"type\":\"finding\",\"source\":\"phoenix\",\"timestamp\":\"${timestamp}\",\"entity_id\":\"${entity_id}\",\"severity\":\"high\",\"correlation_id\":\"${correlation_id}\",\"payload\":{\"finding_type\":\"healing_action\",\"scenario_id\":\"sim-${run_id}\",\"outcome\":\"verified_recovery\",\"description\":\"Phoenix simulator verified service recovery\",\"provenance\":\"simulator\",\"domain\":\"simulator\",\"seed\":42,\"approval_required\":false,\"approval_reason\":\"bounded simulator action\"}}" >/dev/null +phoenix_published_ms="$(now_ms)" echo "==> Verifying the Sentinel incident" verified=false +incident_file="${log_dir}/verified-incident.json" for _attempt in $(seq 1 45); do - if curl --fail --silent --max-time 25 http://127.0.0.1:8090/overview | jq -e --arg correlation_id "${correlation_id}" '.incidents[] | select(.correlation_id == $correlation_id and .sources == ["argus","phoenix"] and .provenance == ["replayed","simulator"])' >/dev/null; then + if curl --fail --silent --max-time 25 http://127.0.0.1:8090/overview | jq -e --arg correlation_id "${correlation_id}" '.incidents[] | select(.correlation_id == $correlation_id and .sources == ["argus","phoenix"] and .provenance == ["replayed","simulator"])' >"${incident_file}"; then verified=true break fi sleep 1 done [[ "${verified}" == "true" ]] || fail "Sentinel did not expose correlation ${correlation_id}." +verified_ms="$(now_ms)" + +sentinel_health="$(curl --fail --silent --max-time 5 http://127.0.0.1:8090/health)" +openai_configured="$(jq -r '.openai_configured == true' <<<"${sentinel_health}")" +mkdir -p "${artifact_dir}" +run_json="${artifact_dir}/${correlation_id}.json" +run_markdown="${artifact_dir}/${correlation_id}.md" +jq -n \ + --arg run_id "${run_id}" \ + --arg generated_at "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ + --arg correlation_id "${correlation_id}" \ + --arg entity_id "${entity_id}" \ + --argjson openai_configured "${openai_configured}" \ + --argjson argus_publish_ms "$((argus_published_ms - demo_started_ms))" \ + --argjson phoenix_publish_ms "$((phoenix_published_ms - argus_published_ms))" \ + --argjson correlation_ms "$((verified_ms - phoenix_published_ms))" \ + --argjson lifecycle_ms "$((verified_ms - demo_started_ms))" \ + --slurpfile incident "${incident_file}" \ + '{ + schema_version: "1.0", verdict: "PASS", run_id: $run_id, generated_at: $generated_at, + correlation: {id: $correlation_id, verified: true, sources: ["argus", "phoenix"], incident: $incident[0]}, + entity: {id: $entity_id}, + timings: { + argus_evidence_publish_ms: $argus_publish_ms, + phoenix_recovery_publish_ms: $phoenix_publish_ms, + sentinel_correlation_ms: $correlation_ms, + total_lifecycle_ms: $lifecycle_ms + }, + recovery: {result: "verified_recovery", availability: "recovery verified; availability percentage not measured"}, + governance: {approval_required: false, reason: "bounded simulator action"}, + openai: {configured: $openai_configured, briefing_generated: false}, + evidence: {provenance: ["replayed", "simulator"], seed: 42, live_chaos: false}, + interpretation: "Sentinel correlated replayed Argus security evidence with a Phoenix simulator recovery outcome for the same resource. This proves the cross-agent control path; it does not claim live threat latency, live chaos execution, or measured 100% availability." + }' >"${run_json}" + +bash "${argus_root}/scripts/render-demo-report.sh" "${run_json}" "${run_markdown}" +cp "${run_json}" "${artifact_dir}/latest-demo.json" +cp "${run_markdown}" "${artifact_dir}/latest-demo.md" + +echo "" +echo "PLATFORM RESILIENCE PROOF: PASS" +printf ' Argus evidence published: %s ms\n' "$((argus_published_ms - demo_started_ms))" +printf ' Phoenix recovery published: %s ms\n' "$((phoenix_published_ms - argus_published_ms))" +printf ' Sentinel correlation: %s ms\n' "$((verified_ms - phoenix_published_ms))" +printf ' Total verified lifecycle: %s ms\n' "$((verified_ms - demo_started_ms))" +echo " Recovery: verified" +echo " Availability: not measured (no false 100% claim)" +echo " Human approval: not required — bounded simulator action" +echo " OpenAI briefing: $([[ "${openai_configured}" == "true" ]] && echo 'configured; not invoked' || echo 'not configured')" +echo " Sources: Argus + Phoenix" +echo " Provenance: replayed + simulator (seed 42)" +echo " JSON evidence: ${artifact_dir}/latest-demo.json" +echo " Markdown evidence: ${artifact_dir}/latest-demo.md" echo "" echo "✅ Full platform demo is ready" echo " Argus: http://127.0.0.1:5173" diff --git a/scripts/platform-local-feed.sh b/scripts/platform-local-feed.sh new file mode 100755 index 0000000..1145f48 --- /dev/null +++ b/scripts/platform-local-feed.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +sog_url="${SOG_URL:-http://127.0.0.1:8010}" +seed="${DEMO_SEED:-42}" +sequence=0 +services=("service/storefront/checkout" "service/payments/payment-api" "service/orders/order-api") + +while true; do + sleep "${DEMO_FEED_INTERVAL_SECONDS:-20}" + sequence=$((sequence + 1)) + entity_id="${services[$((sequence % ${#services[@]}))]}" + timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + correlation_id="local-stream-${seed}-${sequence}-$(date -u +%H%M%S)" + curl --fail --silent -X POST "${sog_url}/findings" -H 'Content-Type: application/json' \ + -d "$(jq -nc --arg id "argus-${correlation_id}" --arg ts "${timestamp}" --arg entity "${entity_id}" --arg corr "${correlation_id}" --argjson seed "${seed}" '{event_id:$id,type:"finding",source:"argus",timestamp:$ts,entity_id:$entity,severity:"medium",correlation_id:$corr,replayed:true,payload:{assessment:"Seeded runtime anomaly replay",provenance:"replayed",demo_data:true,seed:$seed}}')" >/dev/null + curl --fail --silent -X POST "${sog_url}/findings" -H 'Content-Type: application/json' \ + -d "$(jq -nc --arg id "phoenix-${correlation_id}" --arg ts "${timestamp}" --arg entity "${entity_id}" --arg corr "${correlation_id}" --argjson seed "${seed}" '{event_id:$id,type:"finding",source:"phoenix",timestamp:$ts,entity_id:$entity,severity:"medium",correlation_id:$corr,payload:{outcome:"verified_recovery",description:"Synthetic health check passed after bounded recovery",provenance:"simulator",domain:"simulator",demo_data:true,seed:$seed}}')" >/dev/null +done diff --git a/scripts/render-demo-report.sh b/scripts/render-demo-report.sh new file mode 100755 index 0000000..616daa4 --- /dev/null +++ b/scripts/render-demo-report.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +report_json="${1:?usage: render-demo-report.sh REPORT.json REPORT.md}" +report_markdown="${2:?usage: render-demo-report.sh REPORT.json REPORT.md}" + +jq -e ' + .schema_version == "1.0" and + .verdict == "PASS" and + .correlation.verified == true and + .correlation.sources == ["argus", "phoenix"] and + (.evidence.provenance == ["replayed", "simulator"] or + .evidence.provenance == ["live_chaos", "observed"]) and + .recovery.result == "verified_recovery" +' "${report_json}" >/dev/null + +jq -r ' + "# Sentinel Platform Resilience Proof: \(.verdict)\n", + "**Run:** `\(.run_id)` ", + "**Correlation:** `\(.correlation.id)` ", + "**Generated:** \(.generated_at) ", + "**Mode:** \(if .evidence.live_chaos then "observed security evidence + live Chaos Mesh" else "deterministic replay + simulator (no live Chaos Mesh)" end)\n", + "## Evidence scorecard\n", + "| Measurement | Result |", + "|---|---|", + "| \(if .evidence.live_chaos then "Argus observed-detection wait" else "Argus evidence published" end) | \(if .evidence.live_chaos then .timings.argus_detection_ms else .timings.argus_evidence_publish_ms end) ms |", + "| \(if .evidence.live_chaos then "Phoenix verified recovery" else "Phoenix recovery published" end) | \(if .evidence.live_chaos then .timings.recovery_ms else .timings.phoenix_recovery_publish_ms end) ms |", + "| Sentinel correlation | \(.timings.sentinel_correlation_ms) ms |", + "| Total verified lifecycle | \(.timings.total_lifecycle_ms) ms |", + "| Recovery | \(.recovery.result) |", + "| Availability claim | \(.recovery.availability) |", + "| Human approval | \(if .governance.approval_required then "required" else "not required" end) — \(.governance.reason) |", + "| OpenAI | \(if .openai.configured then "configured; briefing not invoked by this proof" else "not configured; deterministic proof still valid" end) |", + "| Sources | \(.correlation.sources | join(" + ")) |", + "| Provenance | \(.evidence.provenance | join(" + ")) |", + "| Experiment | \(if .evidence.live_chaos then .evidence.scenario_id else ("seed " + (.evidence.seed | tostring)) end) |\n", + "## Interpretation\n", + .interpretation, + "\n## Judge path\n", + "1. Open Sentinel at .", + "2. Open **Incidents** and select `\(.correlation.id)`.", + "3. Follow the Argus evidence into the verified Phoenix recovery outcome." +' "${report_json}" >"${report_markdown}" diff --git a/scripts/seed-platform-local.sh b/scripts/seed-platform-local.sh new file mode 100755 index 0000000..e0fc28a --- /dev/null +++ b/scripts/seed-platform-local.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +sog_url="${SOG_URL:-http://127.0.0.1:8010}" +argus_url="${ARGUS_URL:-http://127.0.0.1:8000}" +seed="${DEMO_SEED:-42}" +run_id="${DEMO_RUN_ID:-local-$(date -u +%Y%m%dT%H%M%SZ)}" +iso_offset() { python3 -c 'import datetime,sys; print((datetime.datetime.now(datetime.timezone.utc)+datetime.timedelta(seconds=int(sys.argv[1]))).isoformat().replace("+00:00","Z"))' "$1"; } + +post_json() { + curl --fail --silent --show-error -X POST "${sog_url}$1" \ + -H 'Content-Type: application/json' -d "$2" >/dev/null +} + +entities='[ + {"entity_id":"node/demo/control-plane","entity_type":"node","name":"demo-control-plane","namespace":"cluster","labels":{"role":"control-plane","demo-data":"synthetic"}}, + {"entity_id":"node/demo/worker-a","entity_type":"node","name":"demo-worker-a","namespace":"cluster","labels":{"role":"worker","zone":"a","demo-data":"synthetic"}}, + {"entity_id":"node/demo/worker-b","entity_type":"node","name":"demo-worker-b","namespace":"cluster","labels":{"role":"worker","zone":"b","demo-data":"synthetic"}}, + {"entity_id":"service/edge/gateway","entity_type":"service","name":"gateway","namespace":"edge","labels":{"tier":"frontend","demo-data":"synthetic"},"slo_target":0.9995}, + {"entity_id":"service/storefront/web","entity_type":"service","name":"web","namespace":"storefront","labels":{"tier":"frontend","demo-data":"synthetic"}}, + {"entity_id":"service/storefront/checkout","entity_type":"service","name":"checkout","namespace":"storefront","labels":{"tier":"api","demo-data":"synthetic"},"slo_target":0.9999}, + {"entity_id":"service/payments/payment-api","entity_type":"service","name":"payment-api","namespace":"payments","labels":{"tier":"api","demo-data":"synthetic"},"slo_target":0.9999}, + {"entity_id":"service/payments/fraud-engine","entity_type":"service","name":"fraud-engine","namespace":"payments","labels":{"tier":"worker","demo-data":"synthetic"}}, + {"entity_id":"service/orders/order-api","entity_type":"service","name":"order-api","namespace":"orders","labels":{"tier":"api","demo-data":"synthetic"}}, + {"entity_id":"service/orders/order-worker","entity_type":"service","name":"order-worker","namespace":"orders","labels":{"tier":"worker","demo-data":"synthetic"}}, + {"entity_id":"service/data/postgres-primary","entity_type":"service","name":"postgres-primary","namespace":"data","labels":{"tier":"database","demo-data":"synthetic"}}, + {"entity_id":"service/data/redis-cache","entity_type":"service","name":"redis-cache","namespace":"data","labels":{"tier":"cache","demo-data":"synthetic"}}, + {"entity_id":"service/observability/prometheus","entity_type":"service","name":"prometheus","namespace":"observability","labels":{"tier":"monitoring","demo-data":"synthetic"}}, + {"entity_id":"service/argus-system/argus-agent","entity_type":"service","name":"argus-agent","namespace":"argus-system","labels":{"tier":"security","demo-data":"synthetic"}}, + {"entity_id":"service/phoenix-system/phoenix-agent","entity_type":"service","name":"phoenix-agent","namespace":"phoenix-system","labels":{"tier":"resilience","demo-data":"synthetic"}}, + {"entity_id":"pod/storefront/checkout-a","entity_type":"pod","name":"checkout-a","namespace":"storefront","labels":{"app":"checkout","demo-data":"synthetic"}}, + {"entity_id":"pod/payments/payment-api-a","entity_type":"pod","name":"payment-api-a","namespace":"payments","labels":{"app":"payment-api","demo-data":"synthetic"}}, + {"entity_id":"pod/orders/order-api-a","entity_type":"pod","name":"order-api-a","namespace":"orders","labels":{"app":"order-api","demo-data":"synthetic"}} +]' + +while IFS= read -r entity; do post_json /entities "${entity}"; done < <(jq -c '.[]' <<<"${entities}") + +edges='[ + ["service/edge/gateway","service/storefront/web","depends-on",0.8], + ["service/storefront/web","service/storefront/checkout","depends-on",0.9], + ["service/storefront/checkout","service/payments/payment-api","depends-on",1.0], + ["service/storefront/checkout","service/orders/order-api","depends-on",0.9], + ["service/payments/payment-api","service/payments/fraud-engine","depends-on",0.7], + ["service/payments/payment-api","service/data/postgres-primary","depends-on",1.0], + ["service/payments/fraud-engine","service/data/redis-cache","depends-on",0.6], + ["service/orders/order-api","service/orders/order-worker","depends-on",0.8], + ["service/orders/order-worker","service/data/postgres-primary","depends-on",0.9], + ["service/storefront/checkout","service/data/redis-cache","depends-on",0.8], + ["service/observability/prometheus","service/payments/payment-api","depends-on",0.4], + ["service/argus-system/argus-agent","service/edge/gateway","depends-on",0.5], + ["service/phoenix-system/phoenix-agent","service/storefront/checkout","depends-on",0.5], + ["pod/storefront/checkout-a","node/demo/worker-a","runs-on",1.0], + ["pod/payments/payment-api-a","node/demo/worker-b","runs-on",1.0], + ["pod/orders/order-api-a","node/demo/worker-a","runs-on",1.0] +]' +while IFS= read -r edge; do + post_json /edges "$(jq -nc --argjson row "${edge}" '{source_id:$row[0],target_id:$row[1],edge_type:$row[2],weight:$row[3]}')" +done < <(jq -c '.[]' <<<"${edges}") + +curl --fail --silent --show-error -X POST "${argus_url}/simulate-threats" \ + -H 'Content-Type: application/json' \ + -d "{\"count\":12,\"scenario\":\"mixed\",\"seed\":${seed}}" >/dev/null + +timestamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +for case_spec in \ + "checkout-egress|service/storefront/checkout|critical|Blocked suspicious egress from checkout|Network latency recovered within SLO" \ + "payment-shell|service/payments/payment-api|high|Unexpected shell detected in payment API|Payment replica replaced and health verified" \ + "orders-secret|service/orders/order-api|medium|Sensitive configuration read observed|Order API policy restored and verified"; do + IFS='|' read -r case_name entity_id severity argus_summary phoenix_summary <<<"${case_spec}" + correlation_id="local-${run_id}-${case_name}" + post_json /findings "$(jq -nc --arg id "argus-${correlation_id}" --arg ts "${timestamp}" --arg entity "${entity_id}" --arg severity "${severity}" --arg corr "${correlation_id}" --arg summary "${argus_summary}" --argjson seed "${seed}" '{event_id:$id,type:"finding",source:"argus",timestamp:$ts,entity_id:$entity,severity:$severity,correlation_id:$corr,replayed:true,payload:{finding_type:"falco_alert",assessment:$summary,provenance:"replayed",demo_data:true,seed:$seed}}')" + post_json /findings "$(jq -nc --arg id "phoenix-${correlation_id}" --arg ts "${timestamp}" --arg entity "${entity_id}" --arg corr "${correlation_id}" --arg summary "${phoenix_summary}" --arg healthy "$(iso_offset 0)" --arg fault "$(iso_offset 1)" --arg detected "$(iso_offset 2)" --arg decision "$(iso_offset 3)" --arg approval "$(iso_offset 4)" --arg recovered "$(iso_offset 5)" --arg verified "$(iso_offset 6)" --argjson seed "${seed}" '{event_id:$id,type:"finding",source:"phoenix",timestamp:$ts,entity_id:$entity,severity:"high",correlation_id:$corr,payload:{finding_type:"healing_action",outcome:"verified_recovery",description:$summary,provenance:"simulator",domain:"simulator",demo_data:true,approval_required:false,seed:$seed,evidence_source:"Synthetic simulator + replay fixture",metrics:{detection_ms:1000,recovery_ms:1000,availability_percent:"not measured"},lifecycle:[{stage:"healthy",timestamp:$healthy,source:"synthetic fixture",evidence:"Synthetic service baseline established",status:"simulated"},{stage:"fault_injected",timestamp:$fault,source:"Phoenix simulator",evidence:"Synthetic fault rule activated; no cluster mutation",status:"simulated"},{stage:"detection",timestamp:$detected,source:"replayed Argus",evidence:"Replayed evaluation finding published",status:"replayed"},{stage:"decision",timestamp:$decision,source:"Sentinel simulator policy",evidence:"Bounded simulator recovery selected",status:"simulated"},{stage:"human_approval",timestamp:$approval,source:"governance policy",evidence:"Approval not required for bounded simulator action",status:"not required"},{stage:"recovery",timestamp:$recovered,source:"Phoenix simulator",evidence:$summary,status:"simulated"},{stage:"verification",timestamp:$verified,source:"synthetic health check",evidence:"Simulator outcome verified; availability not measured",status:"simulated"}]}}')" +done + +echo "${run_id}" diff --git a/scripts/tests/test-demo-doctor.sh b/scripts/tests/test-demo-doctor.sh new file mode 100644 index 0000000..3b17015 --- /dev/null +++ b/scripts/tests/test-demo-doctor.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +doctor="${repo_root}/scripts/demo-doctor.sh" + +fail() { echo "FAIL: $*" >&2; exit 1; } +assert_contains() { grep -Fq -- "$1" "${doctor}" || fail "missing $2"; } + +bash -n "${doctor}" +bash "${doctor}" --help | grep -Fq 'This command is read-only.' || fail "help does not state mutation boundary" +if bash "${doctor}" invalid >/dev/null 2>&1; then fail "invalid mode was accepted"; fi + +assert_contains 'OPENAI_API_KEY is configured (value not displayed)' "secret-safe credential check" +assert_contains 'LIVE_DEMO_DRY_RUN=true' "live dry-run delegation" +assert_contains 'Next: make demo-platform' "portable next action" +assert_contains 'Next: make demo-platform-live' "live next action" +assert_contains 'INJECT LIVE FAULT' "live approval reminder" + +echo "Judge demo doctor tests passed." diff --git a/scripts/tests/test-demo-platform-live.sh b/scripts/tests/test-demo-platform-live.sh new file mode 100644 index 0000000..50cdc12 --- /dev/null +++ b/scripts/tests/test-demo-platform-live.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -Eeuo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +driver="${repo_root}/scripts/demo-platform-live-proof.sh" + +fail() { echo "FAIL: $*" >&2; exit 1; } +assert_contains() { + local pattern="$1" description="$2" + grep -Fq -- "${pattern}" "${driver}" || fail "missing ${description}" +} + +bash -n "${driver}" +assert_contains 'LIVE_DEMO_DRY_RUN' "non-mutating dry-run" +assert_contains 'INJECT LIVE FAULT' "explicit fault approval" +assert_contains 'LIVE_DEMO_CONTEXT' "exact context approval" +assert_contains 'fault_seconds <= 30' "bounded fault duration" +assert_contains 'sentinel-live-demo' "isolated default namespace" +assert_contains 'replicas: 2' "redundant disposable target" +assert_contains 'mode:"one"' "single-replica blast radius" +assert_contains 'get podchaos "${backend_ref}"' "real Chaos Mesh object verification" +assert_contains 'replacement_ready' "replacement readiness verification" +assert_contains 'measured_availability_percent' "measured availability evidence" +assert_contains 'provenance:"observed"' "observed Argus provenance" +assert_contains 'provenance:"live_chaos"' "live Phoenix provenance" +assert_contains 'stage:"human_approval"' "human approval lifecycle stage" +assert_contains 'stage:"verification"' "verification lifecycle stage" +assert_contains 'metrics:{detection_ms:' "measurable proof metrics" +assert_contains 'Observed Argus evidence is now present.' "just-in-time approval after detection" +assert_contains 'delete namespace "${namespace}"' "isolated cleanup boundary" + +echo "Live platform demo guard tests passed." diff --git a/scripts/tests/test-demo-platform.sh b/scripts/tests/test-demo-platform.sh index 3465238..624d153 100755 --- a/scripts/tests/test-demo-platform.sh +++ b/scripts/tests/test-demo-platform.sh @@ -3,14 +3,48 @@ set -Eeuo pipefail repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" script="${repo_root}/scripts/demo-platform.sh" +local_script="${repo_root}/scripts/demo-platform-local.sh" +renderer="${repo_root}/scripts/render-demo-report.sh" +tmp_dir="$(mktemp -d "${TMPDIR:-/tmp}/demo-report-test.XXXXXX")" +trap 'rm -rf "${tmp_dir}"' EXIT bash -n "${script}" +bash -n "${local_script}" +bash -n "${repo_root}/scripts/seed-platform-local.sh" +bash -n "${repo_root}/scripts/platform-local-feed.sh" +bash -n "${renderer}" grep -q 'DEMO_PLATFORM_DRY_RUN' "${script}" grep -q 'DEMO_PLATFORM_EXIT_AFTER_READY' "${script}" grep -q 'correlation_id="judge-demo-' "${script}" grep -q 'provenance.*replayed' "${script}" grep -q 'provenance.*simulator' "${script}" +grep -q 'proof.lifecycle.*length.*== 7' "${local_script}" +grep -q 'availability_percent == "not measured"' "${local_script}" grep -q 'Press Ctrl-C to stop only processes started by this command' "${script}" +grep -q 'PLATFORM RESILIENCE PROOF: PASS' "${script}" +grep -q 'availability percentage not measured' "${script}" +grep -q 'sentinel_ui_ok' "${script}" +grep -q 'listener_cwd.*expected_cwd' "${script}" +grep -q '5175/api/health' "${script}" +grep -q 'Kubernetes API calls: 0' "${local_script}" +grep -q 'PHOENIX_LOCAL_DEMO=true' "${local_script}" +grep -q 'synthetic cluster topology' "${local_script}" +grep -q 'docker run --rm' "${local_script}" +grep -q 'demo-data.*synthetic' "${repo_root}/scripts/seed-platform-local.sh" + +jq -n '{ + schema_version: "1.0", verdict: "PASS", run_id: "test", generated_at: "2026-07-18T00:00:00Z", + correlation: {id: "judge-demo-test", verified: true, sources: ["argus", "phoenix"]}, + timings: {argus_evidence_publish_ms: 10, phoenix_recovery_publish_ms: 20, sentinel_correlation_ms: 30, total_lifecycle_ms: 60}, + recovery: {result: "verified_recovery", availability: "recovery verified; availability percentage not measured"}, + governance: {approval_required: false, reason: "bounded simulator action"}, + openai: {configured: true, briefing_generated: false}, + evidence: {provenance: ["replayed", "simulator"], seed: 42}, + interpretation: "Deterministic test proof." +}' >"${tmp_dir}/report.json" +bash "${renderer}" "${tmp_dir}/report.json" "${tmp_dir}/report.md" +grep -q 'Sentinel Platform Resilience Proof: PASS' "${tmp_dir}/report.md" +grep -q 'availability percentage not measured' "${tmp_dir}/report.md" echo "demo-platform orchestration tests passed" diff --git a/setup.md b/setup.md index 17d0871..b12633b 100644 --- a/setup.md +++ b/setup.md @@ -1,10 +1,74 @@ -# Argus — Local Setup +# Sentinel Platform — Judge Setup and Argus Cluster Guide -This guide covers everything needed to run Argus locally: spinning up the cluster, deploying the security stack, and running the console. +This guide has one entry point. Choose exactly one path, run its read-only doctor, then +run the command it prints. The portable path is recommended for judges; the live path is +the measured k3s proof used for the technical demo. + +## Start here — choose one path + +The repositories must share this layout: + +```text +Projects/ +├── argus-k8s/ # run every command below here +└── sentinel-stack/ + ├── phoenix/ + ├── sentinel/ + └── sentinel-platform/ +``` + +### Path A — portable full-platform demo (recommended) + +This path needs Docker or OrbStack but does not need Kubernetes. It runs the real local +Argus, Phoenix, Sentinel, and SOG services against clearly labeled synthetic topology, +replayed Argus evidence, and Phoenix simulator outcomes. + +```bash +make doctor +make demo-platform +``` + +Open the three consoles: + +- Argus: +- Phoenix: +- Sentinel: + +Successful runs write `artifacts/demo-platform/latest-demo.{json,md}`. `Ctrl-C` stops +only command-owned processes and removes the disposable Redis container. + +### Path B — guarded live k3s proof + +This path needs the existing three-node `argus` k3s cluster and deployed security, +chaos, agent, and SOG stack. The doctor is read-only: + +```bash +kubectl config use-context argus +make doctor-live +make demo-platform-live +``` + +The live command separately asks for the exact context and `INJECT LIVE FAULT`. It +creates only `sentinel-live-demo`, targets one of two disposable replicas, waits for +observed Argus evidence, verifies the real Phoenix/Chaos Mesh object and Kubernetes +replacement, measures HTTP availability and recovery time, and requires Sentinel +correlation. Successful runs write +`artifacts/demo-platform/latest-live-demo.{json,md}`. `Ctrl-C` deletes only the isolated +namespace. + +### What the modes claim + +| Mode | Evidence provenance | Kubernetes mutation | Availability claim | +|---|---|---:|---| +| Portable | `replayed` + `simulator` | None | Explicitly not measured | +| Live k3s | `observed` + `live_chaos` | Isolated, approved namespace only | Measured by continuous HTTP probes | + +The remainder of this document explains how the maintained Argus k3s environment is +constructed and troubleshot. Judges using Path A can stop here. --- -## Prerequisites +## Cluster prerequisites **Hardware:** macOS on Apple Silicon (M1/M2/M3). The cluster runs three ARM64 Linux VMs.