ateom: add GetWorkloadStats RPC and retain actor attribution - #667
ateom: add GetWorkloadStats RPC and retain actor attribution#667Tim Bai (baizhenyu) wants to merge 1 commit into
Conversation
60c4024 to
59407c2
Compare
| // between "available" and "executing", so it is safe to call on a timer for | ||
| // the whole lifetime of a workload. Returns FAILED_PRECONDITION when the | ||
| // ateom is "available" (nothing to measure). | ||
| rpc StatsWorkload(StatsWorkloadRequest) returns (StatsWorkloadResponse) {} |
There was a problem hiding this comment.
I would recommend to name it GetWorkloadStats (and rename the request and response as well) to follow the AIP standard: https://google.aip.dev/131
There was a problem hiding this comment.
Done — GetWorkloadStats, GetWorkloadStatsRequest, GetWorkloadStatsResponse. Agreed that StatsWorkload read as if "Stats" were a verb.
One note on how far I took AIP-131: §3.1 of docs/api-style-guide.md also says the response must be the resource itself rather than a ...Response wrapper, which would make this returns (WorkloadStats). I kept the Request/Response pair as you wrote it, since the rest of this internal service (RunWorkload, CheckpointWorkload, RestoreWorkload) is paired that way and a sample is not really a addressable resource. Happy to go the other way if you prefer the strict reading.
| string actor_template_name = 5; | ||
|
|
||
| // "gvisor" | "microvm". | ||
| string sandbox_class = 6; |
There was a problem hiding this comment.
Should we make it an enum?
There was a problem hiding this comment.
Done — SandboxClass enum with SANDBOX_CLASS_UNSPECIFIED / _GVISOR / _MICROVM, following §5.1 of the style guide (prefixed values, _UNSPECIFIED zero) and matching the existing SnapshotScope in this file. The values mirror the ate.dev/v1alpha1 SandboxClass the WorkerPool and ActorTemplate are configured with.
| // as though they were. | ||
| // | ||
| // Empty when no measurement was taken. | ||
| string source = 7; |
There was a problem hiding this comment.
Same, should we make the source an enum?
There was a problem hiding this comment.
Done, same treatment — StatsSource with STATS_SOURCE_UNSPECIFIED / _CGROUP / _GUEST_AGENT. _UNSPECIFIED carries the "no measurement was taken" meaning the empty string used to, so the "all four measurements are zero" case is still distinguishable from a real zero.
| // The workload is gone and the ateom is back to "available": there is | ||
| // nothing left to measure, and holding the identity would let a later | ||
| // StatsWorkload report a checkpointed actor as though it were still running. | ||
| s.activeActor = nil |
There was a problem hiding this comment.
Is it too late to set nil here? What if listSnapshotFiles failed above?
There was a problem hiding this comment.
Good catch — that was a real bug, fixed.
listSnapshotFiles can return early, and by that point cmdCheckpoint / cmdFsCheckpoint has already destroyed the sandbox, so the ateom would have kept advertising a checkpointed actor as live for whatever GetWorkloadStats calls arrived before atelet recycled the worker.
Moved the clear to immediately after the snapshot switch, which is the exact moment the sandbox stops existing. Everything below it (cleanupContainersAfterCheckpoint, UnmountAllUnder, CleanupActorNetwork, listSnapshotFiles) is bookkeeping over a dead sandbox. Deliberately nothing above it clears: a checkpoint that failed may well have left the workload running, and reporting its usage is the honest answer there.
I checked the micro-VM CheckpointWorkload for the same shape and it does not have it — the guest is only Paused until teardownActor, so every early return leaves a live VM still in s.running, and teardownActor / delete(s.running, ...) are adjacent with nothing fallible in between.
7d3b0a3 to
e591f8b
Compare
Adds ateom.Ateom/GetWorkloadStats, the RPC atelet will poll for per-actor
resource usage, plus the attribution retention both ateom runtimes need to
label a sample. The measurement half of each runtime lands in the two
follow-ups; GetWorkloadStats returns Unimplemented until then, so this change
puts nothing half-populated on the wire.
GetWorkloadStats is a pure read: unlike Run/Checkpoint/Restore it does not
move the ateom between "available" and "executing", so it is safe to call on
a timer for a workload's whole lifetime. The request carries the actor UID
the caller believes is executing here, so a recycled worker is rejected with
FAILED_PRECONDITION rather than reporting a different actor's numbers under
the requested actor's name.
sandbox_class and source are enums (SandboxClass, StatsSource) rather than
strings: both are closed sets the ateom binary picks from, and a typo in a
free-form string would silently split a metric in two downstream.
Neither runtime kept the actor's identifying fields past the call that
started the workload -- they arrive on Run/Restore and nothing downstream
needed them. A usage sample is only useful once attributed, so both now hold
them for as long as they are executing:
* ateom-gvisor gains AteomService.activeActor, set by RunWorkload and
RestoreWorkload and cleared by CheckpointWorkload and by both boot-failure
paths, tracking exactly the available/executing state machine.
* ateom-microvm gains runningActor.activeActor, populated from the existing
actorBootParams on both the cold-boot and restore paths; the existing
delete from s.running in teardownActor clears it.
The extraction from the request is shared in internal/ateomstats since both
binaries need it. The type is ActorAttribution, not ActorIdentity: "actor
identity" already means a credential in this repo (ateapi's ActorIdentity
service, substratex509, ateompath.ActorIdentityDirPath), and nothing here is
a secret or is presented as proof of anything.
Part of agent-substrate#594
e591f8b to
3e687b3
Compare
Phase 0 of #550, tracked by #594. First of three stacked PRs.
Adds
ateom.Ateom/GetWorkloadStats, the RPC atelet will poll for per-actor resourceusage, plus the actor attribution both ateom runtimes need to label a sample. The measurement half of each runtime lands in the two follow-ups;
GetWorkloadStatsreturnsUnimplementeduntil then, so this change puts nothinghalf-populated on the wire.
The RPC
GetWorkloadStatsis a pure read: unlikeRunWorkload/CheckpointWorkload/RestoreWorkloadit does not move the ateom between "available" and "executing",so it is safe to call on a timer for a workload's whole lifetime.
The request carries the actor UID the caller believes is executing here. A worker
can be recycled between atelet's view of the world and the call landing, so a
mismatch is rejected with
FAILED_PRECONDITIONrather than reporting a differentactor's numbers under the requested actor's identity.
FAILED_PRECONDITIONisalso the answer when the ateom is available — there is nothing to measure.
The response is one sample, measured at sandbox granularity (which today
equals the actor; per-container attribution would need the gVisor sentry's own
accounting). It echoes the measured actor's identity so the caller can attribute
the sample without holding its own worker→actor mapping, and carries
sandbox_classandsourceso two differently-measured numbers are not silentlycompared as though they were the same thing. Both are enums — closed sets the
ateom binary picks from, where a typo in a free-form string would silently split
a metric in two downstream.
Attribution retention
Neither runtime kept the actor's identifying fields past the call that started the
workload — they arrive on Run/Restore and nothing downstream needed them. A usage
sample is only useful once attributed, so both now hold them for as long as they
are executing:
AteomService.activeActor, set byRunWorkloadandRestoreWorkload, cleared byCheckpointWorkloadand by both boot-failurepaths — tracking exactly the available/executing state machine on the service.
runningActor.activeActor, populated from the existingactorBootParamson both the cold-boot and restore paths; the existing deletefrom
s.runninginteardownActorclears it.The extraction from the request is shared in
internal/ateomstats, since bothbinaries need the same mapping.
ActorAttributioncomposes the existingresources.ActorRefrather than repeating its two fields, so the actor'sAtespacestays visibly attached to the actor — worth noting becauseTemplateNamespacebeside it is an unrelated namespace (a Kubernetes one;ActorTemplateis a namespaced CRD, while an atespace is Substrate's owntenancy unit).
It is deliberately not called
ActorIdentity: "actor identity" already meansa credential in this repo — the
ateapi.ActorIdentityservice,substratex509,ateompath.ActorIdentityDirPath, and #670 building on all three. Nothing in thistype is a secret or is presented as proof of anything; it is the tuple a usage
sample is labeled with.
Reviewer notes
activeActorhas no reader in this PR. It is written and cleared but neverread, because both
GetWorkloadStatsbodies are still stubs. The readers arrivewith the measurement in the two follow-ups; the stub comments point there.
identity, but that is a sample, not a datapoint. Per the decisions in Observability At-Scale #174,
actor/atespace identity belongs in logs and traces rather than as TSDB labels,
and only template-level dimensions become metric labels. That conversion is
atelet's, in Phase 1/2.
cpu_usage_usecresets on restore. A restore recreates the sandbox, so thecounter restarts at zero while the actor UID stays the same. Documented on the
field; callers computing deltas must treat a decrease as a reset. If that proves
too subtle in practice, an explicit epoch marker is a backward-compatible
addition later.
workerpool_nameis deliberately absent. atelet does not know it today;adding the field later is backward compatible.
Testing
internal/ateomstats—TestActorAttributionFromRequestpins the mapping fromboth request types, including the empty and nil cases (the callers are not
defensive about the request pointer, so the nil-safety of the generated getters
is load-bearing). Five deliberately distinct placeholder values, so a field
wired to the wrong source is visible.
cmd/ateom-microvm—TestActorBootParamsAttributionpins theactorBootParams→ActorAttributionmapping, andTestActorBootParamsAttributionMatchesRequestchecks that the two hops (request→ boot params → attribution, written in different files) compose back into what
the caller sent.
TestGetWorkloadStatsUnimplementedpins the stub's advertisedcontract, and
TestAteomServiceStartsAvailablechecks a fresh gVisor serviceretains no attribution (a non-nil zero value there would make an idle ateom
report an empty actor's usage instead of refusing).
RunWorkload,RestoreWorkloadandCheckpointWorkloadeach reach for netlink, runsc and theworker pod's netns within a few lines of entry, so there is no seam to drive
them from
go test. Noted incmd/ateom-gvisor/stats_test.gorather thancovered with a fake; the transitions get verified end to end once
GetWorkloadStatsreturns real data.Both ateom packages are
//go:build linux, so their tests do not execute ondarwin. They were run natively on a Linux host in addition to the local
compile-check, along with
hack/verify/shellcheck.sh(which needs docker).Still ahead
/sys/fs/cgroup/pause, notmain/: every application process runs inside the sentry, which is a singleprocess in
pause/.StatsContainerover the existingvsock connection, not the host cgroup (guest RAM is a fixed allocation, so the
host cgroup barely moves with the workload). Closes Phase 0: StatsWorkload RPC on ateom (proto, both runtime reads, identity retention) #594.