Skip to content

ateom: add GetWorkloadStats RPC and retain actor attribution - #667

Open
Tim Bai (baizhenyu) wants to merge 1 commit into
agent-substrate:mainfrom
baizhenyu:stats-workload-proto
Open

ateom: add GetWorkloadStats RPC and retain actor attribution#667
Tim Bai (baizhenyu) wants to merge 1 commit into
agent-substrate:mainfrom
baizhenyu:stats-workload-proto

Conversation

@baizhenyu

@baizhenyu Tim Bai (baizhenyu) commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Phase 0 of #550, tracked by #594. First of three stacked PRs.

Adds ateom.Ateom/GetWorkloadStats, the RPC atelet will poll for per-actor resource
usage, plus the actor attribution both ateom runtimes need to label a sample. The measurement half of each runtime lands in the two follow-ups;
GetWorkloadStats returns Unimplemented until then, so this change puts nothing
half-populated on the wire.

The RPC

GetWorkloadStats is a pure read: unlike RunWorkload / CheckpointWorkload /
RestoreWorkload it does not move the ateom between "available" and "executing",
so it is safe to call on a timer for a workload's whole lifetime.

The request carries the actor UID the caller believes is executing here. A worker
can be recycled between atelet's view of the world and the call landing, so a
mismatch is rejected with FAILED_PRECONDITION rather than reporting a different
actor's numbers under the requested actor's identity. FAILED_PRECONDITION is
also the answer when the ateom is available — there is nothing to measure.

The response is one sample, measured at sandbox granularity (which today
equals the actor; per-container attribution would need the gVisor sentry's own
accounting). It echoes the measured actor's identity so the caller can attribute
the sample without holding its own worker→actor mapping, and carries
sandbox_class and source so two differently-measured numbers are not silently
compared as though they were the same thing. Both are enums — closed sets the
ateom binary picks from, where a typo in a free-form string would silently split
a metric in two downstream.

Attribution retention

Neither runtime kept the actor's identifying fields past the call that started the
workload — they arrive on Run/Restore and nothing downstream needed them. A usage
sample is only useful once attributed, so both now hold them for as long as they
are executing:

  • ateom-gvisor gains AteomService.activeActor, set by RunWorkload and
    RestoreWorkload, cleared by CheckpointWorkload and by both boot-failure
    paths — tracking exactly the available/executing state machine on the service.
  • ateom-microvm gains runningActor.activeActor, populated from the existing
    actorBootParams on both the cold-boot and restore paths; the existing delete
    from s.running in teardownActor clears it.

The extraction from the request is shared in internal/ateomstats, since both
binaries need the same mapping. ActorAttribution composes the existing
resources.ActorRef rather than repeating its two fields, so the actor's
Atespace stays visibly attached to the actor — worth noting because
TemplateNamespace beside it is an unrelated namespace (a Kubernetes one;
ActorTemplate is a namespaced CRD, while an atespace is Substrate's own
tenancy unit).

It is deliberately not called ActorIdentity: "actor identity" already means
a credential in this repo — the ateapi.ActorIdentity service, substratex509,
ateompath.ActorIdentityDirPath, and #670 building on all three. Nothing in this
type is a secret or is presented as proof of anything; it is the tuple a usage
sample is labeled with.

Reviewer notes

  • activeActor has no reader in this PR. It is written and cleared but never
    read, because both GetWorkloadStats bodies are still stubs. The readers arrive
    with the measurement in the two follow-ups; the stub comments point there.
  • No metric labels are added here. The response carries actor and atespace
    identity, but that is a sample, not a datapoint. Per the decisions in Observability At-Scale #174,
    actor/atespace identity belongs in logs and traces rather than as TSDB labels,
    and only template-level dimensions become metric labels. That conversion is
    atelet's, in Phase 1/2.
  • cpu_usage_usec resets on restore. A restore recreates the sandbox, so the
    counter restarts at zero while the actor UID stays the same. Documented on the
    field; callers computing deltas must treat a decrease as a reset. If that proves
    too subtle in practice, an explicit epoch marker is a backward-compatible
    addition later.
  • workerpool_name is deliberately absent. atelet does not know it today;
    adding the field later is backward compatible.

Testing

  • internal/ateomstatsTestActorAttributionFromRequest pins the mapping from
    both request types, including the empty and nil cases (the callers are not
    defensive about the request pointer, so the nil-safety of the generated getters
    is load-bearing). Five deliberately distinct placeholder values, so a field
    wired to the wrong source is visible.
  • cmd/ateom-microvmTestActorBootParamsAttribution pins the
    actorBootParamsActorAttribution mapping, and
    TestActorBootParamsAttributionMatchesRequest checks that the two hops (request
    → boot params → attribution, written in different files) compose back into what
    the caller sent.
  • Both runtimes — TestGetWorkloadStatsUnimplemented pins the stub's advertised
    contract, and TestAteomServiceStartsAvailable checks a fresh gVisor service
    retains no attribution (a non-nil zero value there would make an idle ateom
    report an empty actor's usage instead of refusing).
  • Not unit-tested: the gVisor set/clear transitions. RunWorkload,
    RestoreWorkload and CheckpointWorkload each reach for netlink, runsc and the
    worker pod's netns within a few lines of entry, so there is no seam to drive
    them from go test. Noted in cmd/ateom-gvisor/stats_test.go rather than
    covered with a fake; the transitions get verified end to end once
    GetWorkloadStats returns real data.

Both ateom packages are //go:build linux, so their tests do not execute on
darwin. They were run natively on a Linux host in addition to the local
compile-check, along with hack/verify/shellcheck.sh (which needs docker).

Still ahead

  • PR 2 — gVisor: read the sandbox cgroup. Reads /sys/fs/cgroup/pause, not
    main/: every application process runs inside the sentry, which is a single
    process in pause/.
  • PR 3 — micro-VM: read the guest agent's StatsContainer over the existing
    vsock connection, not the host cgroup (guest RAM is a fixed allocation, so the
    host cgroup barely moves with the workload). Closes Phase 0: StatsWorkload RPC on ateom (proto, both runtime reads, identity retention) #594.

Comment thread internal/proto/ateompb/ateom.proto Outdated
// between "available" and "executing", so it is safe to call on a timer for
// the whole lifetime of a workload. Returns FAILED_PRECONDITION when the
// ateom is "available" (nothing to measure).
rpc StatsWorkload(StatsWorkloadRequest) returns (StatsWorkloadResponse) {}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would recommend to name it GetWorkloadStats (and rename the request and response as well) to follow the AIP standard: https://google.aip.dev/131

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — GetWorkloadStats, GetWorkloadStatsRequest, GetWorkloadStatsResponse. Agreed that StatsWorkload read as if "Stats" were a verb.

One note on how far I took AIP-131: §3.1 of docs/api-style-guide.md also says the response must be the resource itself rather than a ...Response wrapper, which would make this returns (WorkloadStats). I kept the Request/Response pair as you wrote it, since the rest of this internal service (RunWorkload, CheckpointWorkload, RestoreWorkload) is paired that way and a sample is not really a addressable resource. Happy to go the other way if you prefer the strict reading.

Comment thread internal/proto/ateompb/ateom.proto Outdated
string actor_template_name = 5;

// "gvisor" | "microvm".
string sandbox_class = 6;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make it an enum?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — SandboxClass enum with SANDBOX_CLASS_UNSPECIFIED / _GVISOR / _MICROVM, following §5.1 of the style guide (prefixed values, _UNSPECIFIED zero) and matching the existing SnapshotScope in this file. The values mirror the ate.dev/v1alpha1 SandboxClass the WorkerPool and ActorTemplate are configured with.

Comment thread internal/proto/ateompb/ateom.proto Outdated
// as though they were.
//
// Empty when no measurement was taken.
string source = 7;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same, should we make the source an enum?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, same treatment — StatsSource with STATS_SOURCE_UNSPECIFIED / _CGROUP / _GUEST_AGENT. _UNSPECIFIED carries the "no measurement was taken" meaning the empty string used to, so the "all four measurements are zero" case is still distinguishable from a real zero.

Comment thread cmd/ateom-gvisor/main.go Outdated
// The workload is gone and the ateom is back to "available": there is
// nothing left to measure, and holding the identity would let a later
// StatsWorkload report a checkpointed actor as though it were still running.
s.activeActor = nil

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it too late to set nil here? What if listSnapshotFiles failed above?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — that was a real bug, fixed.

listSnapshotFiles can return early, and by that point cmdCheckpoint / cmdFsCheckpoint has already destroyed the sandbox, so the ateom would have kept advertising a checkpointed actor as live for whatever GetWorkloadStats calls arrived before atelet recycled the worker.

Moved the clear to immediately after the snapshot switch, which is the exact moment the sandbox stops existing. Everything below it (cleanupContainersAfterCheckpoint, UnmountAllUnder, CleanupActorNetwork, listSnapshotFiles) is bookkeeping over a dead sandbox. Deliberately nothing above it clears: a checkpoint that failed may well have left the workload running, and reporting its usage is the honest answer there.

I checked the micro-VM CheckpointWorkload for the same shape and it does not have it — the guest is only Paused until teardownActor, so every early return leaves a live VM still in s.running, and teardownActor / delete(s.running, ...) are adjacent with nothing fallible in between.

@baizhenyu
Tim Bai (baizhenyu) force-pushed the stats-workload-proto branch 2 times, most recently from 7d3b0a3 to e591f8b Compare July 31, 2026 20:17
Adds ateom.Ateom/GetWorkloadStats, the RPC atelet will poll for per-actor
resource usage, plus the attribution retention both ateom runtimes need to
label a sample. The measurement half of each runtime lands in the two
follow-ups; GetWorkloadStats returns Unimplemented until then, so this change
puts nothing half-populated on the wire.

GetWorkloadStats is a pure read: unlike Run/Checkpoint/Restore it does not
move the ateom between "available" and "executing", so it is safe to call on
a timer for a workload's whole lifetime. The request carries the actor UID
the caller believes is executing here, so a recycled worker is rejected with
FAILED_PRECONDITION rather than reporting a different actor's numbers under
the requested actor's name.

sandbox_class and source are enums (SandboxClass, StatsSource) rather than
strings: both are closed sets the ateom binary picks from, and a typo in a
free-form string would silently split a metric in two downstream.

Neither runtime kept the actor's identifying fields past the call that
started the workload -- they arrive on Run/Restore and nothing downstream
needed them. A usage sample is only useful once attributed, so both now hold
them for as long as they are executing:

  * ateom-gvisor gains AteomService.activeActor, set by RunWorkload and
    RestoreWorkload and cleared by CheckpointWorkload and by both boot-failure
    paths, tracking exactly the available/executing state machine.
  * ateom-microvm gains runningActor.activeActor, populated from the existing
    actorBootParams on both the cold-boot and restore paths; the existing
    delete from s.running in teardownActor clears it.

The extraction from the request is shared in internal/ateomstats since both
binaries need it. The type is ActorAttribution, not ActorIdentity: "actor
identity" already means a credential in this repo (ateapi's ActorIdentity
service, substratex509, ateompath.ActorIdentityDirPath), and nothing here is
a secret or is presented as proof of anything.

Part of agent-substrate#594
@baizhenyu Tim Bai (baizhenyu) changed the title ateom: add StatsWorkload RPC and retain actor identity ateom: add StatsWorkload RPC and retain actor attribution Jul 31, 2026
@baizhenyu Tim Bai (baizhenyu) changed the title ateom: add StatsWorkload RPC and retain actor attribution ateom: add GetWorkloadStats RPC and retain actor attribution Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Phase 0: StatsWorkload RPC on ateom (proto, both runtime reads, identity retention)

2 participants