Table of contents
Polyad (named after polyads in mathematics) is a Kubernetes operator and Python SDK for writing and orchestrating adaptive microservices. Deploy, connect and scale applications as graphs within and across clusters, enforce Cheeger bounds, and adjust topology, traffic and capacity to application demand. Compose batch jobs, persistent services and supporting resources into reusable Graphs and PolyGraphs. Define how work starts, how components communicate, how they respond to application demand and which constraints must hold as the application changes, from a workflow in one cluster to a hierarchy spanning multiple clusters.[2] The Kubernetes operator is built on Kopf, the Kubernetes Operators Framework for Python.
Get started · Documentation · Helm chart
- Compose workloads and resources. Run finite pipelines and persistent services using Jobs, Deployments, StatefulSets or DaemonSets, with dependencies, activation policies, storage and placement.
- Scale within graph constraints. KEDA and ReplicaGroups scale individual services, whole graphs or nested compositions. Polyad refreshes live graph state and enforces GraphPolicies, including size, shape and structural Cheeger bounds, before applying scaling changes.
- Adapt to administrator-defined demand. Soul searching uses offered throughput or a named signal, such as queued jobs, to select approved Cheeger targets, connection layouts, traffic splits and capacity lookahead. Observe recommendations or allow bounded adaptation, including preparation before throughput falls behind. Hard rules and resource ceilings stay fixed.
- Make connectivity explicit. Choose replica connection patterns, including custom edges, and enforce network boundaries with optional NetworkPolicy and Istio integration. Istio percentage routing divides requests among workload or graph replicas using fixed splits, demand tiers or measured spare capacity.
- Let services participate. Through the standalone Python SDK, workloads can submit compositions, activate work, request TTL-bound connections and discover permitted services across the atlas, with filtered event hooks and peer-approved connections. This application model is Service Symbiosis: services with different roles discover compatible peers and adjust their work together. Shared types and JSON Schemas are also available separately from the operator.
- Rebalance event subscriptions. Subscribe over SSE or WebSocket and opt into paced reconnections as operator replicas change. Clients resume from their checkpoints through Service/Istio routing or discovered Pod IPs.
- Coordinate across clusters. A root operator can run in a dedicated management cluster, deploy graphs and execution replicas into registered workload clusters, and collect their observations centrally.
- Choose the control-plane layout. Select a single-replica or HA Helm profile. HA can run dense operators or separate gateway, executor and telemetry components managed through the operator's own Graph. Optionally persist graph state and tracked measurements in PostgreSQL.
- Observe and coordinate changes. Inspect Prometheus and JSON metrics and OpenTelemetry traces and decision logs. The write pipeline coalesces duplicate mutations, checks dependencies before dispatch and returns stale decisions to reconciliation.
Deploy the Kubernetes operator with the Helm chart, or try the local Python scheduler. The examples cover pipelines, services, spot work, storage and nested graphs.
For application feedback, start with the approved load-profile example
and its Helm reference values.
The demand guide explains signal names,
units and thresholds; use Observe mode to inspect recommendations before enabling adaptation.
Applications can install the Python SDK or just the
shared types without installing the operator. From a
checkout, use pip install ./pkg/polyad-types; the standalone distribution is
named polyad-types and exposes polyad_types.
Use Service Symbiosis to write
producers and consumers that cooperate across Graphs and PolyGraphs: discover
compatible peers, react to connection and capacity deltas, propagate backpressure
and report useful completion. Run the standalone python demo/soul.py
local demonstration to watch three service
processes share queued work over an added TCP connection and roll their child
workers under load. It compares completion time, latency and backlog against a
fixed chain with the same load and worker limits.
Run python demo/nature.py for the parent Natural Selection demonstration:
changing requirements select capabilities and routes, mutate services, preserve
useful survivors and retire excluded processes.
Explore graph concepts, graph policies, composition requests, the composition API, networking and event subscriptions, temporary connections, API keys and shared request lanes, or the development guide. See the documentation index for lifecycle, status, health and configuration references.
While at Klaviyo, I briefly crossed paths with engineers working on a project converting the company's cloud architecture to a setup where clusters managed other clusters, a concept they'd designed at Medium. That idea helped motivate Polyad's root control plane and nested PolyGraphs. Medium's Kubernetes Infrastructure At Medium provides public background on this strategy's multi-cluster infrastructure, gradual rollouts and capacity planning.
Deploying a distributed application means deciding how its services connect, which work can run together, and how those relationships should change as demand grows. Those relationships need to remain understandable and enforceable as the application spans more services, teams and clusters.
A scaling decision that helps one service can leave the rest of its pipeline overloaded or disconnected. Teams need a way to express what must remain true for the whole application as its parts grow, shrink or span more clusters. Polyad aims to make that coordination repeatable, with room for applications to adapt within boundaries their administrators can trust.
Growth puts pressure on both request latency and recovery time, especially when it adds interconnected dependencies and increases resource utilization. A request involving more services has more opportunities to wait on a slow component, and shared resources accumulate queues. Dean and Barroso's The Tail at Scale explains how occasional delays can become a dominant performance problem at larger scales.
Recovery can also become slower and more involved. An overloaded service can shift work onto its neighbors, while retries add more demand to struggling dependencies. Restoring useful capacity may require several services to recover, new Pods or nodes to become ready, and caches to warm. Google's account of cascading failures describes how these effects can reinforce one another.
Growth also creates opportunities for parallelism and spare capacity. Dividing work into well-defined boundaries lets individual requests and recovery decisions remain local as the application expands. Polyad's architectural objective is to preserve these short reaction paths while coordinating changes that affect shared dependencies. Available capacity becomes useful when services can reach it and adapt how they use it.
Service Symbiosis brings part of that response into the microservices themselves. A producer can reduce outstanding work when a consumer slows down, or send compatible work to an authorized peer with spare capacity. A consumer can adjust concurrency, switch worker profiles or drain accepted jobs before replacing workers. The Python SDK's adaptive interface exposes connection, capacity and decision changes to application-defined strategies. These local responses let services use their existing resources while additional infrastructure is being prepared. Services can also request temporary connections and new compositions within their permissions.
Graphs and PolyGraphs make related workloads and their connections reusable deployment units. GraphPolicies express their structural requirements, which Polyad checks against live state before applying ReplicaGroup scaling requests. A root control plane extends that model across registered clusters, coordinating deployments and collecting observations.
At those larger boundaries, Soul searching uses observed demand, completed throughput and reported spare capacity to guide approved connection changes and traffic balancing. Traffic can shift between individual services, whole graph replicas or PolyGraph replicas as their measured ability to accept work changes. Cheeger bounds constrain structural bottlenecks, while capacity preparation gives upcoming stages and node autoscalers notice of future demand. Application measurements and load tests establish useful targets for each boundary.
The same approach applies to a small producer-consumer pair and an application spread across clusters: respond close to the work, report what happened, and coordinate broader changes where dependencies are shared. Immediate admission and backpressure stay local; graph changes follow configured observation windows, cooldowns and resource limits. This gives a growing system ways to respond promptly without every reaction waiting for coordination across the whole application. Its adaptation envelope records which changes it can absorb, how quickly it recovers and which constraints it must preserve.
A graph groups related work and describes how its parts depend on each other. Its nodes can be tasks, services, resources or other graphs. A data pipeline might fetch records, process partitions in parallel, then publish the results; a service graph might keep consumers and their supporting resources running.[3]
Compose smaller workflows into an application with PolyGraph. Each child
reports progress to its parent, giving the root a combined view of the work.[4]
In the diagrams below, green marks work and graph summaries, amber marks constraints or recurrence, and gray marks resources and containing boundaries.
Example: nested graphs reporting to an application root
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart BT
job["Workload"] --> batch["Graph · batch"]
daemon["Daemon"] --> service["Graph · service"]
spot["Graph · spot work"] --> group["PolyGraph · processing"]
batch --> group
group --> root["PolyGraph · application"]
service --> root
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef constraint fill:#fff3d6,stroke:#926000,color:#513900
class job,daemon,batch,service,group,root execution
class spot constraint
Read about graphs of graphs.
Build workflows from reusable definitions and trace each instance to its
Kubernetes resources. GraphPolicy lets engineers constrain what users can
schedule by size, shape, nesting and mathematical properties.[5][6]
Example: reusable graph definitions with structural constraints
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart LR
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef constraint fill:#fff3d6,stroke:#926000,color:#513900
classDef resource fill:#eeeeee,stroke:#777777,color:#444444
request["Composition request<br/>IDs and references"]:::resource
rules["GraphPolicy<br/>size, shape, spectrum"]:::constraint
root["PolyGraph root<br/>aggregate status"]:::execution
request --> root
rules -. "constrains each boundary" .-> root
subgraph left["Graph instance: left"]
a["Workload instance"]:::execution
end
subgraph right["Graph instance: right"]
b["Workload instance"]:::execution
end
root --> a
root --> b
definition["Shared graph definition"]:::resource
definition -. "instantiates" .-> a
definition -. "instantiates" .-> b
Read about composition requests and GraphPolicy constraints.
Application code can participate in adaptation, using available capacity while protecting work it has already accepted. The Soul study demonstrates this with six real Python service processes, a load generator and a monitoring parent. Each service uses the SDK's adaptation strategies to respond to changing demand and controlled disturbances.
The parent routes new jobs toward services with room to accept them. Services pause new assignments when observations become unavailable, peers become unhealthy or connection permission expires, while draining work already accepted. The study compares fixed and adaptive trials under the same offered load and resource ceilings, verifies every completed job, and records queue sizes, completion latency and process lifecycles.
Build services that put spare capacity to work and release extra workers when demand falls. These recorded process graphs show services adding batch workers, switching to compact workers under modeled memory pressure and recovering their original footprint. Accepted jobs remain tracked through each transition. Click the figure to inspect the worker identities and connections.
The Nature study adds composition decisions above those local adaptations. When the required output changes, its Natural Selection planner chooses which service implementations and connections can deliver it within the configured process and cost limits.
Let services contribute in different ways as requirements change. A and F switch to implementations that perform both processing steps; B and E keep running and connect to new services that supply the second step. C and H retire after draining their accepted work. When the original requirement returns, the planner restores the original composition. Each selected service continues adapting its own workers. Click the figure to follow the process identities and routes.
For developers, the reusable pattern is to keep business processing separate from
the policies that decide when to accept work, which worker profile to run and how
to replace it safely. Extend AdaptiveService,
choose strategies for your application's constraints,
and measure whether those adaptations improve useful completion and recovery.
The study's strategy modules and
repeatable run instructions provide a working starting
point with before, during and after measurements.
Running workloads can submit their next graph, read its status and subscribe to graph events through the operator's optional APIs. Services route requests to ready replicas; explicitly authorized callers can connect from other namespaces.[15]
A running service can also pulse downstream workloads or daemon replica groups, with explicit concurrency and frequency policies.[17]
The Python SDK supports filtered event hooks over SSE or WebSocket. With optional connection rebalancing, operators send paced reconnect instructions, called copulses, when membership changes or an administrator starts a roll. Clients retain their last completed checkpoint, refresh ready endpoints and reconnect through Service/Istio routing or direct client-side round robin. Graceful shutdown gives subscriptions a bounded window to reconnect. See the scale-out and scale-down sequence diagrams and typed Helm reference.
With a capacity policy, Polyad forecasts upcoming stages while earlier work runs, giving a compatible node autoscaler advance notice. Dependencies and gates still decide when the next stage starts.[16]
Example: API access, event streams and advance capacity requests
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart TB
subgraph local["Workloads · operator namespace"]
caller["Managed workload"]
end
subgraph remote["Workloads · another namespace"]
subscriber["Authorized workload"]
end
access["Explicit network access<br/>Optional Istio identity authorization"]
composition["Composition Service · 8090<br/>Submit graphs and read status"]
events["Event Service · 8091<br/>Subscribe to graph observations"]
subgraph control["Node group · operators"]
operator["Ready operator replicas<br/>APIs and graph scheduling"]
cache[("Shared Dragonfly cache<br/>Queues and event history")]
end
demand["Kubernetes API<br/>ProvisioningRequest or placeholder Pods"]
autoscaler["Node autoscaler"]
machines["Target worker node group<br/>Capacity for upcoming stages"]
caller -->|"Authenticated requests"| access
subscriber -->|"Cross-namespace requests"| access
access --> composition
access -->|"GET /v1/events"| events
composition --> operator
events --> operator
operator <-->|"Coordination and observations"| cache
events -. "SSE or WebSocket observations" .-> local
events -. "SSE or WebSocket observations" .-> remote
operator -->|"Forecast before next stage"| demand
demand -->|"Upcoming resource demand"| autoscaler
autoscaler -->|"Provision nodes when supported"| machines
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef constraint fill:#ffe3a3,stroke:#926000,color:#513900
classDef resource fill:#eeeeee,stroke:#777777,color:#444444
class caller,subscriber execution
class operator,access constraint
class composition,events,cache,demand,autoscaler,machines resource
style local fill:#ffffff,stroke:#667085,stroke-width:2px,color:#344054
style remote fill:#e2e6ec,stroke:#667085,stroke-width:2px,color:#344054
style control fill:#fff3d6,stroke:#926000,stroke-width:2px,color:#513900
linkStyle default stroke:#475467,stroke-width:2px
Read about workload API access, event subscriptions, and advance capacity requests.
KEDA can autoscale different levels of the hierarchy
by targeting a ReplicaGroup's Kubernetes /scale subresource. The group's
template determines what each additional replica creates:
- Daemon replicas: another service instance, backed by its selected Deployment
or StatefulSet. Set the Daemon's own
replicas: 1when each group copy should represent one desired Pod. - Graph replicas: another complete workflow or service graph, including its workloads, resources and internal connections.
- PolyGraph or nested ReplicaGroup replicas: another composition of graphs or replica groups, allowing scaling at multiple levels of the same application.
For example, scale a worker pool inside a processing graph as its queue grows, and scale copies of the whole processing graph as demand for complete pipelines grows. Scale one group instance independently, or scale a shared definition to update every inheriting instance; see instance and shared scaling.
Before creating or retiring copies, Polyad refreshes the owning graph family's topology and checks replica bounds and applicable GraphPolicies, including structural limits and Cheeger constraints. KEDA supplies the requested count; constraints can block its application. Target the ReplicaGroup to use these checks: directly autoscaling a generated Deployment or StatefulSet bypasses graph admission. See constraints before scaling.
When percentage routing is configured, a positive traffic assignment also blocks removing its destination. Drain its share to zero before scale-in; newly added copies need explicit routing assignments. See traffic balancing and scaling.
A ReplicaGroup of PolyGraphs can also scale a complete cross-cluster composition. Each destination can independently scale its own local groups. The multicluster scaling diagram shows where each cluster refreshes live values and enforces its own rules.
Soul searching lets a Graph or PolyGraph respond
to application demand within administrator-approved profiles. Demand defaults to
offered work per second. Administrators can instead
select an exact signal name and unit (such
as queueDepth in jobs or activeSessions in sessions) and define the thresholds
that select each profile. The complete resource example
includes the Graph, its GraphPolicy and its Workload/Daemon definitions. An authorized
application reporter supplies the measurements; configuring a signal does not
automatically scrape it.
A profile combines a separate application Cheeger target with optional
traffic percentages and capacity preparation
settings. Observe reports recommendations; Adapt may apply approved changes.
The default Shortfall trigger waits for completed throughput to fall behind.
The optional Demand trigger allows preparation while throughput still keeps up,
or while queued work awaits processing.
For example, sustained growth past an approved queue-depth threshold can select a denser connection layout, rebalance traffic and prepare three dependency stages ahead instead of one. Fresh samples, stabilization, cooldowns and change budgets govern those adjustments. Every change must satisfy live GraphPolicies and the fixed graph and operator capacity ceilings.
| Control | Responsibility |
|---|---|
| Soul searching | Select approved connection, traffic and preparation settings from demand |
| KEDA/HPA | Request replica counts for the configured scaling target |
| Polyad capacity planner | Prepare known upcoming execution nodes and coordinate workload admission |
| Kubernetes and the node autoscaler | Schedule Pods and provision machines for scheduling demand |
Soul searching works independently of KEDA and leaves replica counts to the existing scaling controller. Lookahead prepares upcoming work already described by the graph; it does not add spare application replicas to an entirely deployed service. Actual node provisioning still depends on the configured autoscaler.
See profile configuration and limits, the preparation sequence diagram, and the runnable example. For the distinction between hard structural bounds and application targets, see comparing Cheeger controls.
Each ReplicaGroup can choose its own connection mode. Here, one subgraph connects whole graph replicas in a Ring; another combines daemon replicas in a Star with bidirectional connections and a FullMesh. Edges between enclosing graphs and groups are declared separately at their network boundaries.
Example: three replica layouts and connections across subgraphs
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart TB
subgraph application["PolyGraph · application"]
direction TB
subgraph processing["Graph · processing"]
subgraph pipelines["pipelines · Ring"]
direction LR
p0["replica-0<br/>Graph instance"]
p1["replica-1<br/>Graph instance"]
p2["replica-2<br/>Graph instance"]
p0 --> p1 --> p2 --> p0
end
end
subgraph serving["Graph · serving"]
direction TB
subgraph routers["routers · Star"]
direction LR
s0["replica-0<br/>Daemon · hub"]
s1["replica-1<br/>Daemon"]
s2["replica-2<br/>Daemon"]
s3["replica-3<br/>Daemon"]
s0 <--> s1
s0 <--> s2
s0 <--> s3
end
subgraph caches["caches · FullMesh"]
direction LR
m0["replica-0<br/>Daemon"]
m1["replica-1<br/>Daemon"]
m2["replica-2<br/>Daemon"]
m0 <--> m1
m1 <--> m2
m2 <--> m0
end
routers -->|"TCP 6379"| caches
end
processing -->|"TCP 9000"| serving
end
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef boundary fill:#f2f4f7,stroke:#667085,color:#344054
classDef replicas fill:#ffffff,stroke:#667085,stroke-width:2px,color:#344054
class p0,p1,p2,s0,s1,s2,s3,m0,m1,m2 execution
class processing,serving boundary
class pipelines,routers,caches replicas
style application fill:#ffffff,stroke:#667085,stroke-width:2px,color:#344054
linkStyle default stroke:#475467,stroke-width:2px
The three inner boxes are ReplicaGroups. Their internal edges use these settings:
| ReplicaGroup | Copies | Connection configuration | Internal ports |
|---|---|---|---|
pipelines |
Graph | Ring | TCP 8080 |
routers |
Daemon | Star, bidirectional: true |
TCP 9000 |
caches |
Daemon | FullMesh | TCP 6379 |
Arrows show directed data-flow connections; double arrows declare both directions. The application boundary connects processing to serving on TCP 9000; the serving boundary connects routers to caches on TCP 6379. These connections become transport grants when graph networking is enabled, subject to inherited restrictions. Copies without a configured mode remain Independent, with no inter-copy edges. Scaling rebuilds the selected pattern and notifies workloads through topology events.
This example fits within one cluster. For remote Graph workloads, see the east-west gateway diagram and direct Pod routing diagram. Remote placement and data-flow edges need separately configured traffic policies.
Group workloads into subgraphs with explicit network connections. Scoped rules control traffic across boundaries and namespaces; optional Istio integration adds HTTP and service-identity authorization.[7][8]
Optional percentage routing controls how incoming requests are divided among connected replicas, while network policies control which traffic is permitted. Polyad generates Istio VirtualServices and DestinationRules for local sidecar HTTP, HTTP/2 and gRPC Services. A Graph or PolyGraph copy receives its share through its entrypoint workloads. Try the traffic-balancing example, or compare fixed percentages with the Tiers and Headroom feedback modes.
Example: subgraph connections and cross-namespace authorization
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart TB
rules["GraphPolicy · subtree scope"] -. inherits .-> group
subgraph group["PolyGraph · application"]
direction TB
subgraph producers["Graph · producers"]
task["Workload"] --> sender["Daemon"]
end
subgraph consumers["Graph · consumers"]
receiver["Daemon"] --> report["Workload"]
end
producers -->|"TCP 8080"| consumers
end
external["Service identity · another namespace"] -->|"GET /status"| consumers
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef constraint fill:#fff3d6,stroke:#926000,color:#513900
classDef boundary fill:#f2f4f7,stroke:#667085,color:#344054
class task,sender,receiver,report execution
class rules,external constraint
class group,producers,consumers boundary
Read about network scope and inheritance and cross-namespace authorization. This diagram shows one cluster; remote traffic rules are configured separately at each end of a cross-cluster connection.
Graphs execute within one cluster. PolyGraphs can optionally place child Graphs and nested PolyGraphs in registered remote clusters, composing regions and higher levels. Optional shared observers expose read-only graph snapshots while each cluster's operator enforces its own rules and executes its workloads. See cross-cluster placement, Istio transport and observers.
A root operator control plane can manage this entire hierarchy from a separate management cluster. It installs remote execution replicas, collects their observations centrally and coordinates KEDA through root-local scale targets. See the deployment architecture and complete configuration example.
Example: nested PolyGraphs composing three clusters
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart TB
subgraph east["Cluster east"]
root["PolyGraph<br/>Global application"]
region["PolyGraph<br/>East region"]
batch["Graph · batch<br/>Local Jobs"]
root --> region --> batch
end
subgraph west["Cluster west"]
group["PolyGraph<br/>Western regions"]
service["Graph · service<br/>Local Deployments or StatefulSets"]
group --> service
end
subgraph north["Cluster north"]
analytics["Graph · analytics<br/>Local Jobs and resources"]
end
root -->|"cluster: west"| group
group -->|"cluster: north"| analytics
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
class root,region,batch,group,service,analytics execution
Arrows show declared parent-child ownership; child status rolls back up to the
root. Each Graph's workloads stay inside its cluster. The east operator manages
the western PolyGraph's intent; the west operator manages that PolyGraph's
children, including the Graph in north. Each destination operator executes its
local work. The same hierarchy can be entirely local by omitting cluster.
Read about graphs of graphs, remote placement and ownership, and the execution architecture. The two-cluster example demonstrates local nesting in east with a remote Graph in west; the diagram extends this pattern with another remote PolyGraph and cluster.
Place whole graphs on groups of Kubernetes machines, such as general compute or accelerators. Here, three graphs share two worker groups while coordinated operator replicas and their shared Dragonfly cache run on a third.[9][10]
Example: three graphs across two worker groups
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart TB
subgraph control["Node group · operators"]
operator["Polyad operator replicas"]
cache[("Shared Redis / Dragonfly cache")]
end
api["Kubernetes API"]
operator <-->|"queues and coordination"| cache
operator -->|"ordered resource writes"| api
subgraph groupA["Node group · general compute"]
subgraph graphA["Graph · ingest"]
fetch["Fetch"] --> normalize["Normalize"]
end
subgraph graphB["Graph · publish"]
package["Package"] --> publish["Publish"]
end
end
subgraph groupB["Node group · accelerated compute"]
subgraph graphC["Graph · processing"]
compute["Compute"] --> aggregate["Aggregate"]
end
end
api -->|"graph placement"| graphA
api -->|"graph placement"| graphB
api -->|"graph placement"| graphC
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef constraint fill:#fff3d6,stroke:#926000,color:#513900
classDef resource fill:#eeeeee,stroke:#777777,color:#444444
class fetch,normalize,package,publish,compute,aggregate execution
class operator constraint
class api,cache resource
style control fill:#fff3d6,stroke:#926000,color:#513900
style groupA fill:#eeeeee,stroke:#777777,color:#444444
style groupB fill:#eeeeee,stroke:#777777,color:#444444
Read about graph placement and operator replicas and shared queues.
Express a workflow from preparation to publication, with parallel tasks and gates that wait for a condition or delay. Ordinary Graph placement can select spot capacity; applications choose how to handle interruption and storage.[11][12]
Example: parallel spot workloads behind an admission gate
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart TB
subgraph pipeline["Graph · finite pipeline"]
direction TB
storage[("Resource · shared storage")]
prepare["Workload · prepare"]
gate{"Gate · admission condition"}
subgraph workers["Graph · spot placement"]
direction TB
left["Workload · partition A"]
right["Workload · partition B"]
end
subgraph publish["Graph · publish results"]
merge["Workload · merge"]
report["Workload · report"]
merge -->|completed| report
end
storage -->|ready| prepare
prepare -->|completed| gate
gate -->|allowed| left
gate -->|allowed| right
workers -->|completed| publish
end
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef constraint fill:#ffe3a3,stroke:#926000,color:#513900
classDef resource fill:#eeeeee,stroke:#777777,color:#444444
class prepare,left,right,merge,report execution
class gate constraint
class storage resource
style pipeline fill:#e2e6ec,stroke:#667085,stroke-width:2px,color:#344054
style workers fill:#fff3d6,stroke:#926000,stroke-width:2px,color:#513900
style publish fill:#ffffff,stroke:#667085,stroke-width:2px,color:#344054
linkStyle default stroke:#475467,stroke-width:2px
Read about finite pipelines and admission and delay gates.
Keep services running with Daemon, and repeat finite Graphs with activation
requests. A producer or timer supplies each pulse; the application owns iteration
limits, stop conditions and durable shared state.[13]
Example: persistent services with repeated graph activations
Solid arrows show startup or execution progression; dashed arrows show data flow or a new activation request.
---
config:
theme: base
htmlLabels: false
themeVariables:
primaryTextColor: "#163b29"
secondaryTextColor: "#513900"
tertiaryTextColor: "#344054"
clusterBkg: "#f2f4f7"
clusterBorder: "#667085"
titleColor: "#344054"
edgeLabelBackground: "#f2f4f7"
lineColor: "#667085"
flowchart:
subGraphTitleMargin:
top: 8
bottom: 20
---
flowchart TB
subgraph system["Graph · persistent service"]
direction TB
queue[("Resource · queue service")]
subgraph service["Graph · persistent processing"]
direction LR
ingest["Daemon · ingest"]
process["Daemon · process"]
ingest -->|ready| process
ingest <-.->|"events / acknowledgements"| process
end
subgraph activations["Graph · finite activation target"]
direction LR
sample["Workload · sample"]
adjust["Workload · adjust"]
sample -->|completed| adjust
end
next["Producer / timer pulse"]
activations -->|"completion observed"| next
next -.->|"activation request"| activations
queue -->|ready| service
service -->|ready| activations
end
classDef execution fill:#e3f3e8,stroke:#247047,color:#163b29
classDef recurrence fill:#ffe3a3,stroke:#926000,color:#513900
classDef resource fill:#eeeeee,stroke:#777777,color:#444444
class ingest,process,sample,adjust execution
class next recurrence
class queue resource
style system fill:#e2e6ec,stroke:#667085,stroke-width:2px,color:#344054
style service fill:#ffffff,stroke:#667085,stroke-width:2px,color:#344054
style activations fill:#fff3d6,stroke:#926000,stroke-width:2px,color:#513900
linkStyle default stroke:#475467,stroke-width:2px
Read about repeated execution and activation pulses.
The operator can also request capacity ahead of upcoming stages, helping node autoscalers prepare machines while upstream work runs.
Polyad can run as a compact HA Deployment or manage its own service components
in a Graph. With architecture.mode: Distributed, gateway, executor and telemetry
ReplicaGroups scale independently through KEDA and fresh GraphPolicy checks. A
root bootstrap Deployment retains planning and recovery responsibility. With
root mode enabled, the atlas, a reserved root PolyGraph, contains a Graph for each operator
group. The root operator Graph contains the bootstrap, managed component pipeline,
KEDA and all enabled local chart services; remote operator groups join as peers.
flowchart TB
subgraph operators["Atlas · reserved root PolyGraph"]
subgraph rootGroup["Graph · root operator group"]
subgraph bootstrap["Graph · bootstrap observation"]
root["Helm-owned root Deployment<br/>planning and recovery"]
end
subgraph self["Graph · local service components"]
direction LR
gateway["Gateway replicas<br/>APIs and event subscriptions"]
executor["Executor replicas<br/>graph admission and workloads"]
telemetry["Telemetry replicas<br/>observations and metrics"]
gateway --> executor --> telemetry
end
bootstrap -->|"reconcile and restore components"| self
self -->|"observations"| bootstrap
keda["KEDA Graph<br/>Operator, metrics server and webhooks"]
cache["Dragonfly Graph<br/>Controller, cache and Services"]
pg["PostgreSQL Graph · optional<br/>State and authentication databases"]
support["Other local service Graphs<br/>Endpoints, observer and Istio"]
root <-->|"use and observe"| cache
root <-->|"observe"| support
end
subgraph west["Graph · west workload cluster"]
workers["DaemonSet · execution workers<br/>One Pod per eligible node"]
end
subgraph east["Graph · east workload cluster"]
pool["Deployment · execution workers<br/>KEDA replica scaling"]
end
rootGroup -->|"root coordination"| west
rootGroup -->|"root coordination"| east
workers -->|"observations"| root
pool -->|"observations"| root
end
rules["GraphPolicy<br/>Cheeger ≥ 1; recursive size bound"] -. constrains .-> self
keda -->|"scrape demand"| telemetry
keda -->|"request replica counts"| root
root -. "optional state storage" .-> pg
telemetry -. "persist graph state and parameters" .-> pg
The arrows inside the Graph describe logical stages; actual coordination uses Kubernetes and Dragonfly. Cheeger constrains topology, while queue backlog and HTTP demand drive capacity decisions. See the component and networking diagrams and deployable example.
Adding an OperatorPool in a registered downstream cluster automatically links its Graph into the reserved root PolyGraph. Deployment pools support KEDA replica scaling; DaemonSet capacity follows node eligibility. The root Graph's bootstrap branch observes its existing Deployment, preserving Helm ownership and recovery. The nested component Graph retains its own Cheeger and replica-budget checks. Application event streams exclude this operator tree.
The local service inventory observes the
chart's enabled infrastructure while Helm and upstream operators retain lifecycle
ownership. Set keda.install: true to install the optional pinned KEDA dependency,
or configure references to existing KEDA with the
typed KEDA values.
Administrators can also install downstream workers with Helm and attach their existing Deployments. Helm retains installation and upgrades; each attachment explicitly chooses root/KEDA or downstream replica scaling.
PostgreSQL is optional, disabled by default, and stores graph observations and tracked parameters when enabled. Its optional KEDA configuration scales CloudNativePG instances from operator connection counts, still scraped from the operator. Additional database instances provide standby/read capacity; writes continue through the primary. Encryption-at-rest settings select encrypted storage for managed state and authentication databases, using an administrator-provided StorageClass or GKE disks backed by a Cloud KMS key. Optional record encryption uses an administrator-provided public key to encrypt JSON payloads inside the operator before writing them to either managed or external PostgreSQL databases.
Queue pressure and graph hierarchies are available through the optional Prometheus and JSON metrics API.
Optional OpenTelemetry tracing exports API request,
reconciliation and Kubernetes operation spans to an OTLP/HTTP collector, with
configurable sampling and Secret-backed exporter credentials.
The Helm chart can add Grafana Alloy or Prometheus Agent
to collect component metrics, with Alloy also forwarding container logs and traces.
Decision logs explain
admissions, scaling, topology membership and conflicts, with trace correlation
and independently enabled OTLP log export.
KEDA can scale services or whole graph compositions through
ReplicaGroup, using workload metrics served by the operator.
Argo CD and Flux health checks report graph and leaf health across nested applications.
Polyad coordinates application graphs alongside existing cluster components.
- A general-purpose policy engine such as OPA.
GraphPolicyconstrains the graphs Polyad admits and the resources it compiles. It does not evaluate Rego, replace application authorization, or enforce policy on every Kubernetes API request. Cluster-wide admission policy remains a separate concern.[18][19] - A replacement for the Kubernetes scheduler or node autoscaler. Polyad controls when graph work is admitted and propagates placement constraints. Kubernetes places Pods; the configured autoscaler provisions machines. Grouping work does not guarantee that every Pod starts together.[9][20]
- A service mesh or network transport. Polyad generates network and Istio authorization and routing resources, including optional percentage splits. The cluster's networking implementation and mesh enforce them; drawing a graph connection does not transport application data.[21]
- Automatic process checkpointing or exactly-once execution. Restarting containers with persistent storage requires application recovery logic. Workloads must handle retries and duplicate effects; graph ownership and ordered API writes do not make application operations transactional.[22][10]
GNU General Public License v3.0 only.
Selected background reading for Polyad's architecture and less common design choices.
- Operator framework: Kopf: Kubernetes Operators Framework. The Python framework behind Polyad's Kubernetes watches, startup and shutdown hooks, and health probes. Polyad builds graph reconciliation and coordinated mutations on top of it; see the operator runtime.
- Infrastructure at scale: Eduardo Barth, Medium Engineering, Kubernetes Infrastructure At Medium (February 14, 2023). A case study covering separate clusters, gradual infrastructure rollouts, resource sizing and spare capacity for traffic bursts.
- Responsiveness at scale: Jeffrey Dean and Luiz André Barroso, The Tail at Scale, and Google SRE's Addressing Cascading Failures. How dependency delays, overload and retries affect request latency and recovery, motivating local adaptation and coordination across shared dependencies.
- Structural bottlenecks: Shlomo Hoory, Nathan Linial and Avi Wigderson, Expander Graphs and Their Applications (2006, Section 2.1). The edge-expansion definition used by Polyad's hard Cheeger bounds and throughput-driven Cheeger targets. Administrator-selected demand signals choose calibrated targets; sustained shortfall or the optional Demand trigger can select approved changes within the hard bounds. Optional traffic balancing also uses calibrated splits or per-replica throughput and headroom to redistribute requests. Those routing weights are separate from the unweighted Cheeger value used by both bounds.
- Adaptation envelopes and reachability: Stanford ASL hj_reachability and Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances. Background and numerical backend for the SDK's queue models, reachability guard and local studies.
- Rewriting and composition: Dimitri Ara et al., Polygraphs: From Rewriting to Higher Categories. Background for the rewriting, confluence and higher-dimensional diagrams in the mutation documentation.
- Advance capacity: The Cluster Autoscaler ProvisioningRequest FAQ. Reserving capacity before a workload stage becomes runnable; see Polyad's advance-capacity design.
- Pending delivery recovery: Dragonfly's
XAUTOCLAIM. Recovering unacknowledged stream entries when a queue consumer loses ownership. - GitOps health: Argo CD custom health checks and Flux health expressions. Reporting graph and descendant health to deployment controllers.