vientito

Fleet orchestration for containerized workloads, in R.

A small-fleet application and compute platform. It submits run-to-completion workloads, deploys long-running services, and runs scheduled work across a set of nodes, with a single authoritative controller and a per-node agent driving containerd over its native gRPC API.

Workload state is adjudicated by the controller and observed independently on each node, so a receipt reflects what actually ran rather than what was requested. Nothing in a receipt is copied from the request: the image identity is a chainID read back off the snapshotter, the seccomp verdict is read out of /proc, and a property nobody could read comes back unproven rather than assumed.

Not on CRAN. Not production-ready — see Status.

Install

remotes::install_github("cornball-ai/vientito")

Depends on R >= 4.4.0 and four packages: janssonr, secretbase, grpc, RProtoBuf. Linux only. containerd 2.x is needed to run anything; the controller itself needs nothing but a writable directory.

The controller

Everything the fleet knows lives in an append-only log and is folded into state. Open a directory and it replays; the same records always produce the same fold.

library(vientito)

dir.create(file.path(tempdir(), "fleet"), showWarnings = FALSE)
ctl <- controller_open(file.path(tempdir(), "fleet"))

out <- controller_submit(ctl, "op-1", created_at = 1, now = 2,
                         spec = list(name = "hello",
                                     image = paste0("sha256:", strrep("a", 64)),
                                     args = list("/bin/echo", "hi"),
                                     memory_bytes = 64e6),
                         owner = "troy")
out$disposition
#> [1] "accepted"
out$run$state
#> [1] "pending"

A refusal is an answer, not an error. The call worked and the answer was no, and it comes back as a Run in failed with the reason recorded — because "I submitted that and it was refused" has to be answerable months later.

bad <- controller_submit(ctl, "op-2", 1, 2,
                         list(name = "nope", image = "ubuntu:24.04",
                              args = list("/bin/true")), owner = "troy")
bad$disposition
#> [1] "refused"
bad$reason
#> [1] "the controller has no record of the image ubuntu:24.04; submit a digest until the tag authority holds tags"
bad$run$state
#> [1] "failed"

Every mutating call carries an operation id. Resending one returns the first answer and creates no second effect, so a client that lost its response retries safely.

again <- controller_submit(ctl, "op-1", 1, 3,
                           list(name = "hello",
                                image = paste0("sha256:", strrep("a", 64)),
                                args = list("/bin/echo", "hi"),
                                memory_bytes = 64e6), owner = "troy")
again$disposition
#> [1] "replayed"
identical(again$run$run_id, out$run$run_id)
#> [1] TRUE

Services

Desired state, not commands. A client says how many replicas it wants and the controller reconciles, so a message nobody sent cannot leave the fleet in a state nobody asked for.

spec <- list(name = "web", image = paste0("sha256:", strrep("a", 64)),
             args = list("/usr/bin/python3", "-m", "http.server", "8000"),
             memory_bytes = 256e6, network = "published",
             ports = list(list(name = "http", container_port = 8000L)))

svc <- controller_service_declare(ctl, "op-3", 1, 2, "web", spec, replicas = 2,
                                  owner = "troy")
svc$service$replicas
#> [1] 2

Scaling does not restate the spec, and does not roll anything. A revision is identified by its content, so a caller forced to resend the spec would roll every replica any time it changed the count.

up <- controller_service_update(ctl, "op-4", 3, 4, "web", replicas = 3)
identical(up$service$revision, svc$service$revision)
#> [1] TRUE

Discovery is the only answer to "where is service X" — workloads do not find each other by being on a network together, and there is no in-container DNS.

service_endpoints(ctl, "web", now = 5)
#> list()
is.null(service_endpoints(ctl, "no-such-service", now = 5))
#> [1] TRUE

Those are different answers on purpose. An empty list is worth asking about again; NULL is not.

Scheduled work

Time is data. Occurrences are expanded to instants once, when the schedule is declared or extended, and stored as numbers — so a schedule spanning a daylight-saving transition is testable in a second and means the same thing on every node.

k <- controller_schedule_declare(ctl, "op-5", 5, 6, "nightly",
                                 spec = list(name = "nightly",
                                             image = paste0("sha256:", strrep("a", 64)),
                                             args = list("/bin/true")),
                                 expr = list(kind = "daily", at = "02:00"),
                                 zone = "America/Chicago", owner = "troy")
length(k$schedule$occurrences)
#> [1] 7

There is no default zone. The host's TZ is a deployment accident, and taking it would make one schedule mean different things on different nodes.

An expression that cannot be honoured is refused in front of whoever wrote it, not at 03:00 on a Sunday:

controller_schedule_declare(ctl, "op-6", 5, 6, "monthly-31",
                            spec = list(name = "m",
                                        image = paste0("sha256:", strrep("a", 64)),
                                        args = list("/bin/true")),
                            expr = list(kind = "monthly", day = 31, at = "02:00"),
                            zone = "UTC", owner = "troy")$reason
#> [1] "'monthly' needs a day from 1 to 28; a later day does not exist in every month, and both clamping it and skipping the month would be a schedule that does not do what it says"

The client API

Two gRPC services, split by who calls. Vientito authenticates a tailnet user login against a Principal; NodeControl authenticates a node stable ID against a Node. A client cannot reach node control, so that boundary is structural rather than a permission check somebody has to remember.

The server side is a loop the caller drives, so a controller can interleave serving with its own work:

ctl <- controller_open("/var/lib/vientito")
controller_principal_declare(ctl, "op-p1", 1, 2, "troy", "troy@github",
                             operator = TRUE)

## Bind to the tailnet address, not 0.0.0.0. Authentication resolves the peer's
## address against tailscaled, which is sound only for traffic that arrived over
## the tailnet.
srv <- vientito_serve(ctl, "100.64.0.1:8443")

repeat {
    vientito_poll(srv, timeout_ms = 100L)
    controller_tick(ctl, now = as.numeric(Sys.time()))
}

The client carries no identity. Who the caller is comes from the channel and is resolved by the controller, so there is nothing to configure and nothing a caller can assert about itself:

cl <- vientito_client("100.64.0.1:8443")
now <- as.numeric(Sys.time())

out <- vto_submit(cl, "op-a", now,
                  list(name = "hello",
                       image = paste0("sha256:", strrep("a", 64)),
                       args = list("/bin/echo", "hi"), memory_bytes = 64e6))
out$disposition
#> [1] "accepted"
out$run$owner
#> [1] "troy"

There is no owner parameter on vto_submit, and that is the point — a submission that could name its owner would let anyone attribute work to anyone. It comes back on the Run either way.

Services, schedules and environments have the same shape:

vto_service_declare(cl, "op-b", now, "web", spec, replicas = 2)$disposition
#> [1] "declared"

## Scale to zero is a real request, distinguishable from "leave the count
## alone" — which is why the field is wrapped on the wire.
vto_service_update(cl, "op-c", now, "web", replicas = 0)$service$replicas
#> [1] 0

vapply(vto_service_list(cl), function(s) s$name, character(1))
#> [1] "web"

vto_discover(cl, "web")$found
#> [1] TRUE

found is a field rather than an inference from an empty list: "no such service" and "exists with nothing ready" are different answers, and a caller that cannot tell them apart retries the one that is never coming back.

Why did last night's job fail

A container's output lives on the node that ran it. The controller relays the bytes and stores none of them, so its log stays a record of decisions rather than gaining a sibling store of container chatter.

out <- vto_logs_text(cl, run_id, "stderr")
out$disposition
#> [1] "finished"
cat(out$text)
#> the-reason

Streams are never merged. Given one file both land in it and the order is the runtime's rather than the container's — two independent pipe copies interleave by whichever the shim reaches first — so a merged log would carry a fabricated ordering that reads as evidence later.

Following a running workload is a loop with an offset rather than a stream the controller holds open, so a log is readable when the session stream is not, which is exactly when somebody is trying to find out what went wrong:

chunk <- vto_logs(cl, run_id, "stdout", offset = 0, max_bytes = 4096)
chunk$disposition
#> [1] "bytes"
chunk$next_offset
#> [1] 4096

Send the generation back when you resume. A live container's log is capped by truncating it — nothing else frees the space, because the runtime holds the file open — and truncation moves every byte to a different offset while leaving nothing in the filesystem to say so. An offset alone would then read position N of different content, spliced mid-line, with nothing set to say so:

stale <- vto_logs(cl, run_id, "stdout", offset = 64, generation = 5)
stale$disposition
#> [1] "rotated"
stale$next_offset
#> [1] 0

vto_logs_text does this for you and reports rotated if a cap interrupted it, because what it holds at that point is a log with no visible gap that is missing its middle.

Secrets

Material is delivered as files on a tmpfs and bound read-only into the container. Never environment variables: Spec.process.env reads back verbatim from the container record to anything that can reach the runtime API, which closes that channel before any care about who holds the value matters.

A workload names a secret; it never carries one. There is no field a client could put material in.

vto_submit(cl, "op-d", now,
           list(name = "worker", image = digest,
                args = list("/usr/bin/thing"),
                secrets = list(list(name = "api-key",
                                    mount_path = "/run/secrets/api-key"))))

The Run records the version, not the value. After a compromise, "which runs saw the leaked version" is answerable from the Run alone — a design recording only the name cannot answer it, and one recording the value answers it by making the compromise worse.

run$secrets[[1]]$name
#> [1] "api-key"
run$secrets[[1]]$version_id
#> [1] "v2-63eea86aef337b0d"
run$secrets[[1]]$material
#> NULL

The version is pinned at admission, so a rotation between admission and launch cannot give a Run something other than what its own record says it got.

The receipt attests the mount and never the contents:

Filter(function(e) e$property == "secret_mount", run$receipt$evidence)[[1]]$outcome
#> [1] "proven"

That means the path is a tmpfs and is not backed by persistent storage, read from /proc/<pid>/mountinfo before the workload starts. A secret path mounted from ordinary disk is refuted and the container never runs. What it does not mean is that the content is correct: the agent wrote it, so comparing it back would be the workload attesting itself.

A node only gets a workload carrying a credential if it reports a writable per-user tmpfs and an operator has asserted three things it cannot observe about itself: encrypted swap, hibernation off, crash dumps restricted. Those exist because "tmpfs never reaches a disk" is false — tmpfs pages are ordinary anonymous memory and can be swapped or captured in a hibernation image.

node_assert(ctl, "n-1", swap_encrypted = TRUE, hibernation_disabled = TRUE,
            crashdump_restricted = TRUE, by = "troy")

by is required, and the assertion expires after 90 days. An unattributed claim about a host is indistinguishable from a fact; a two-year-old one is about a machine that may have been reinstalled since. Vientito can check neither, so it asks somebody to look again rather than keep believing an old sentence.

Rotation is not revocation. Adding a version means containers created afterwards receive it; running containers keep what they were given, because nothing can be clawed back. Vientito can stop delivering a version and tell you what holds it. A revoked database password is revoked in the database.

And vientito cannot redact what a workload writes. A job that prints its own credential to stdout has published it, and no filter applied afterwards makes that untrue.

GPUs

A workload asks for a device by the identity the node reports, never by path. Paths get renamed, symlinked and reordered across boots; <class>:<major>:<minor> is what the kernel uses and what the receipt compares.

vapply(node_devices("gpu"), function(d) d$id, character(1))
#> [1] "gpu:195:254" "gpu:509:0" "gpu:509:1" "gpu:195:0" "gpu:195:255"

vto_submit(cl, "op-e", now,
           list(name = "train", image = digest,
                args = list("/usr/bin/train"),
                devices = list(list(id = "gpu:195:0"))))

The driver's control nodes (nvidiactl, nvidia-uvm) are added by the node that runs the workload. Every CUDA process opens them, so claiming one exclusively would mean one GPU workload per machine rather than one per GPU.

A device carries at most one non-terminal Run. A second workload asking for a held GPU stays pending with the holder named, and becomes placeable when that Run reaches a terminal state. There is no lease and no renewal — the Run going terminal is the release, so there is nothing to leak.

placement_candidates(ctl, spec)[[1]]$reason
#> [1] "device gpu:195:0 is held by run r-7621d6b985ea30c4"

The receipt reports what the container's own /dev holds, compared against what was injected:

Filter(function(e) e$property == "devices", run$receipt$evidence)[[1]]$outcome
#> [1] "proven"

It says nothing about exclusivity, and that is deliberate. A device node inside a container proves this container has the device and proves nothing about who else does. Vientito guarantees exclusivity by not placing a second Run against a held device — on the controller, by construction — and a receipt field saying exclusive: true would be a restatement of the request wearing the clothes of a measurement. A process outside vientito that opens the device defeats it silently, and nothing here will detect that.

No nvidia container runtime hook. The agent builds the whole OCI spec and attests it before start, so a prestart hook has nothing to add and would sit inside the attestation window changing what was measured. The node injects the device nodes and binds the driver's userspace libraries itself, and reports its driver version so an image built against a different one is refused rather than crashing.

The library list comes from nvidia-container-cli list where it is installed, because which files a given driver version needs is a database NVIDIA maintains rather than a rule worth reimplementing. Reading that list is not the same dependency as running the hook. A node without the tool falls back to a narrower set and says so.

A CUDA kernel runs in that sandbox — rootless, read-only root, every capability dropped — and the check is the arithmetic rather than the absence of an error:

device NVIDIA GeForce RTX 5060 Ti sm_120
PASS 1048576 elements, 3*1+2 == 5.000000

with and without PTX JIT at load. The fallback glob's 8 paths carry it too: the tool's list is preferred because NVIDIA maintains it per driver version, not because the glob was measured to break CUDA. It was not (docs/findings/gpu-device.md section 8).

A workload claiming several devices can require they share an interconnect:

spec <- list(name = "train", image = digest,
             devices = list(list(id = "gpu:195:0"), list(id = "gpu:195:1")),
             devices_share_group = TRUE)

Groups come from the kernel's device tree — which PCIe bridge each device hangs off — and a node reporting no topology is refused rather than read as agreeing. NVLink is never reported, because no kernel record here names it and the alternative is parsing a table printed for humans. A group labelled nvlink on an unmeasured parse would satisfy exactly the requirement that exists to keep a workload off the wrong hardware.

Nodes also report what they have warm — the images their containerd holds — and placement prefers a warm node among the ones that were feasible anyway:

node_warmth(ag)
#> [1] "image:sha256:d9e853e8..." "image:sha256:1428a953..."

It breaks a tie and does not make one. A warm idle node still loses to a colder node already carrying work, so warmth cannot scatter a binpack, and a warmth report that fails to arrive costs a node nothing. The case it decides is an idle fleet placing its first Run, which is the case where it matters.

Failures are classified rather than flattened, so a caller can tell a refusal from a controller whose local daemon is restarting:

classify <- function(expr) {
    tryCatch(expr,
             vientito_client_denied      = function(e) "refused",
             vientito_client_status      = function(e) "answered no",
             vientito_client_unreachable = function(e) "could not ask")
}

classify(vto_get(cl, "r-nope"))
#> [1] "answered no"
classify(vto_get(vientito_client("127.0.0.1:1"), "r-nope"))
#> [1] "could not ask"

UNAVAILABLE from a controller whose tailscaled is restarting and UNAUTHENTICATED from a peer this fleet does not know arrive at the same call site as a non-OK status with a message, which is exactly why they get different condition classes rather than one every caller is trusted to inspect.

Nodes and receipts

The agent runs unprivileged against rootless containerd. It creates a task, observes it in the pre-start window, and only then starts it — so attestation reads properties of a container that exists and has not run yet.

ag <- agent_open("/run/user/1000/containerd/containerd.sock", "vientito",
                 "/var/lib/vientito/claims")

repeat {
    held <- lapply(claim_ids(store), function(i) list(run_id = i))
    agent_apply(ag, session_open(ctl, "n-1", epoch, claims = held),
                observe = observe, report = report)
}

A receipt reads only from the layer that did the thing:

r <- receipt("vto-r-1",
             list(evidence("image_identity", "proven", source = "snapshotter",
                           observed = "sha256:...", expected = "sha256:..."),
                  evidence("seccomp_policy", "unproven", source = "kernel",
                           note = "the kernel reports a filter and never its rules")),
             require = "image_identity")
r$admitted
#> [1] TRUE
r$unproven
#> [1] "seccomp_policy"

Three outcomes, not two. proven, refuted, and unproven — "no" and "nobody could tell" are different answers, and collapsing them is the receipt's whole point undone.

Design

The design is worked out in the open, in PLAN.md and docs/:

• docs/adr/ — the decisions, each with what it refuses and the test that would catch it being wrong.
• docs/findings/ — what was measured before a mechanism was built. containerd's GC window, tailscale's whois latency, what gRPC hands a server when a client gives up, whether a snapshot survives between Prepare and Containers.Create.
• spike/ — the probes those findings came from, runnable.

Recurring rules that show up throughout:

• A failed observation must never become a confident verdict. An instrument that can return a plausible value where it should error hides its own failure.
• Every mutating call is idempotent on an operation id, so retry is always safe and nothing needs a "did it work?" side channel.
• Nothing is inferred from an absent field. proto3 cannot tell zero from unset, so anything with both a default and a meaningful zero is wrapped.
• A word rather than an enum for a controller's disposition, so a client built against an older build reads an answer it has never seen instead of decoding it to UNSPECIFIED.

Status

Increments 1–9 of PLAN.md are closed: the object and product contracts, the architecture decisions, the walking skeleton on Ubuntu, services and networking, scheduled work, the R environment and image pipeline, the client API, and GPU workloads — the devices section above describes what increment 9 built, including the CUDA kernel measured inside the ordinary sandbox.

3,618 tests offline, plus live arms driven against real containerd and multi-process gRPC. Run the live arms with at_home = TRUE: test_package() defaults to FALSE, which reports them as zero tests and still prints a pass.

Not yet: the Matrix adapter (increment 10) and the immutable-host spike (11). StreamLogs is UNIMPLEMENTED and says so — there is no log transport at all, because open decision #13 has not been made and an empty stream would claim a Run produced no output rather than that nobody can say.

Everything to date has been exercised on a single node. Multi-node placement is implemented and its decisions are tested from records; it has not been run against two real machines.

License

MIT. Copyright cornball.ai.