context_daemon

Theseus Context Daemon

This directory contains the transitional context-management prototype shipped with Theseus: the theseus-contextd daemon, the theseus-context CLI, and their multi-node tests.

Please run one daemon on every node that may host a Theseus communicator.

The daemon is not on the collective data path. Its job is to maintain a node-local, persistent replica of the selection context that Theseus reads when choosing a schedule. The current implementation uses one statically configured primary and a fixed set of secondaries. It implements the same log-oriented mental model, but uses primary-assigned ordering, persistent TCP sessions, and startup snapshot transfer instead of Raft.

This daemon is an ad hoc refactor of what we use to manage node-local persistent contexts during evaluation. It provides no primary election, quorum commit, automatic failover, online membership change, or automatic recovery. In the future, we will rebase the daemon on a etcd-like service for fault tolerance.

For implementation details and guarantees, see DESIGN.md. This README starts with the deployment and runtime mental model, then covers build, configuration, CLI use, failure recovery, and tests.

Mental Model

Selection context and persistent context elements

A selection context is the durable control-plane state used by the Theseus selector. It contains three kinds of elements:

• Attributes describe runtime conditions used by schedule policies. Endogenous attributes are read from supported request hooks; exogenous attributes have versioned values published by external monitors or applications.
• Schedules are immutable schedule JSON files that describe GPU communication work.
• Policies are stored with schedule-manifest entries. A usability expression decides whether a schedule can serve the current request; a preference expression ranks usable schedules; an optional replacement names a structurally related schedule for hot swapping.

The daemon persists this state as ordinary files on every node:

<context_dir>/
├── attribute_manifest.jsonl          # attribute declarations and hooks/log paths
├── schedule_manifest.jsonl           # schedule policies and schedule-file paths
├── attributes/
│   └── <attribute-name>.jsonl        # versioned exogenous values
└── schedules/
    └── <schedule-name>.json          # complete four-level schedule content

Manifest and attribute-log files are append-only JSONL. Schedules are copied into the context before their manifest entries make them visible. The on-disk layout is the only intended coupling between the daemon and the Theseus runtime: the daemon does not call into the runtime, push a selection decision, launch GPU work, or manage a Theseus process.

How Theseus consumes the context

Each node's Theseus runtime reads its local context replica. Point THESEUS_SELECTION_CONTEXT at the same path configured as server.context_dir:

export THESEUS_SELECTION_CONTEXT=/var/lib/theseus/context

Communicators poll these files every THESEUS_CONTEXT_CHECK_PERIOD requests. At an agreement round, the communicator group computes a common visible prefix/version before selecting a schedule. The daemon only replicates context elements, while the runtime's selector agreement determines when an element is safe for all participants to use. There is no synchronization or communication between daemon replication and schedule selection.

How context mutations flow

Users and system components publish context elements through theseus-context:

• new attr declares an endogenous or exogenous attribute;
• set appends a new value to an exogenous attribute log; and
• new sched copies a schedule and appends its policy to the schedule manifest.

The CLI sends one command over a short-lived connection to any selected daemon. A secondary forwards mutations to the primary. The primary serializes mutations, assigns an op_index, applies each entry locally, and replicates it concurrently to all secondaries. Queries are always served from the selected daemon's local state.

flowchart LR
    C[theseus-context]

    subgraph PN["Primary node"]
        P[Primary theseus-contextd]
        PC[(Persistent context)]
        PR[Theseus runtime]
        P -->|"3. build and apply ordered entry"| PC
        PC -.->|"6. runtime observes persistent context"| PR
    end

    subgraph SN["Secondary node(s)"]
        S[Secondary theseus-contextd]
        SC[(Persistent context)]
        SR[Theseus runtime]
        S -->|"5. apply ordered entry"| SC
        SC -.->|"6. runtime observes persistent context"| SR
    end

    C -->|"`1a. client command
(select primary)`"| P
    C -->|"`1b. client command
(select secondary)`"| S
    S -->|"2. forward mutation"| P
    P -->|"4. replicate ordered entry"| S

Loading

For a mutation, accepted means only that the selected daemon placed the command in its bounded in-memory queue. Ordering, persistence, forwarding, and replication continue asynchronously. Use get response LOCAL_ID against that same daemon to retrieve the terminal result. Step 6 is node-local visibility, not a cross-node read barrier.

Every successfully decoded client command receives a random 128-bit local_id from the selected daemon. The response returns node_id and local_id; this pair is the command's identity and is preserved in a forwarded mutation and the resulting ordered entry. Local IDs are not persisted across daemon restarts.

Deployment Lifecycle

The safe startup order is part of the correctness contract:

1. Prepare one configuration per node. Configure exactly one primary and a fixed set of secondaries. Every daemon uses the same absolute server.context_dir, which resolves to node-local storage.

2. Start theseus-contextd on every participating node. The daemons may be launched in any order.

3. Wait until every daemon reports ready. Use the CLI locally on each node or select each endpoint with -address:

   ./bin/theseus-context -config /path/to/config.ini show status

   Reaching the TCP-listening phase is necessary but not sufficient: secondaries must install the primary's startup snapshot, and the primary must issue the final readiness decision. During this phase the primary uses rsync -a --delete, so it may replace or remove files in a secondary's context directory.

4. Only after all daemons are ready, start the Theseus application on every node with THESEUS_SELECTION_CONTEXT set to server.context_dir.

5. While the job runs, submit mutations through any daemon and query their terminal status asynchronously. Do not restart, add, or remove daemons online.

6. Stop the Theseus job before stopping or reconfiguring the daemon set. If any daemon reports inconsistent, follow Recovering an Inconsistent Context before starting another job.

sequenceDiagram
    participant P as Primary daemon
    participant S as Secondary daemons
    participant C as theseus-context CLI
    participant T as Theseus runtimes

    Note over P,S: Start one daemon on every participating node
    P->>S: Startup snapshot (rsync -a --delete)
    S-->>P: Snapshot installed
    P->>S: Final ready decision
    C->>P: show status
    C->>S: show status
    Note over T: Start only after every daemon reports ready
    T->>T: Read node-local persistent context
    C->>P: new attr / set / new sched
    P->>S: Ordered-entry replication
    T->>T: Poll local files and agree on a common visible context

Loading

Build

Run the following commands from the context_daemon directory. Go 1.19 or later is required.

mkdir -p bin
go build -o bin/theseus-contextd ./cmd/theseus-contextd
go build -o bin/theseus-context ./cmd/theseus-context

Configuration

Copy example.ini for each node and adjust the following role-specific settings.

Configure every node with:

• a unique server.node_id;
• server.role set to primary or secondary;
• the same absolute, non-root server.context_dir, using only letters, digits, /, ., _, and -.

Configure the primary with:

• every secondary's daemon TCP endpoint under [secondaries]; and
• the corresponding SSH endpoint under [rsync], using exactly the same node IDs as [secondaries].

Configure each secondary with:

• the primary's node_id, host, and port under [primary]; and
• empty [secondaries] and [rsync] sections.

Startup synchronization requires:

• rsync on the primary and every secondary;
• an SSH server on every secondary;
• non-interactive SSH authentication; and
• an SSH user that can write the secondary's persistent context directory and its parent.

The daemon uses the system SSH host-key and identity configuration.

Runtime Settings

Setting	Requirement or default	Purpose
server.listen_host	0.0.0.0	Address on which the daemon accepts connections.
server.listen_port	55055	Daemon TCP port.
server.log	Optional; stderr when omitted	Runtime log destination. A newly created log file requests mode 0644; an existing file keeps its permissions. The parent directory must already exist.
timeouts.connect	5s	Timeout for establishing a daemon connection.
timeouts.call	30s	Shared timeout for client-command exchanges and daemon calls.
timeouts.startup	2m	Startup synchronization deadline: global across all secondaries on the primary and per incoming synchronization on a secondary.
limits.max_frame_bytes	16777216 (16 MiB)	Maximum framed daemon protocol message size.
mutation.queue_capacity	1024	Maximum number of accepted mutation tasks waiting in the in-memory queue.
mutation.shutdown_timeout	5s	Grace period for accepted mutation work during daemon shutdown before mutation worker cancellation is requested.
mutation.response_timeout	30s	Time after a mutation is accepted at which it is reported as overdue if unfinished.

Start the Daemon Set

Run the following command on every node, using that node's configuration:

THESEUS_DAEMON_CONFIG=/path/to/config.ini ./bin/theseus-contextd

The primary and secondaries may be started in any order. Once the primary starts, every configured secondary must become reachable and complete startup synchronization within the primary's timeouts.startup deadline. The primary retries initial connection attempts, but it does not retry startup synchronization after a connection has been established. The primary remains not_ready until all secondaries are synchronized; each secondary remains not_ready until the primary issues its final readiness decision after all secondaries are synchronized.

Do not start a Theseus runtime until every daemon reports ready: snapshot transfer uses rsync -a --delete and may replace existing persistent context files.

After readiness, the configured daemon set and its daemon sessions are fixed, and restarting a daemon online is unsupported. A daemon that detects a daemon-session loss or daemon-call timeout records a context failure with code daemon_session_lost and fences its persistent context. If the detecting daemon is the primary, it broadcasts the context failure to every reachable secondary. Daemons do not reconnect automatically.

Client Commands

All client commands accept these global flags before the command name:

• -config PATH selects the INI file. The default is $THESEUS_DAEMON_CONFIG, or config.ini when the environment variable is unset.
• -address HOST:PORT selects a daemon endpoint. The default is the local endpoint from the INI file.

Mutation Commands

Command	Purpose
new attr NAME endo TYPE HOOK	Create an endogenous attribute bound to a supported Theseus request hook.
new attr NAME exo TYPE	Create an exogenous attribute whose value can be updated with set.
set NAME VALUE...	Set the next value of an exogenous attribute. int and double require exactly one value; vector requires one or more values.
new sched PATH USABILITY PREFERENCE [REPLACE]	Read a schedule JSON file, attach its usability and preference expressions, and optionally name an existing schedule it replaces.

TYPE is int, double, or vector; vector is persisted as vector<double>.

HOOK identifies a supported endogenous request field, such as request.messagesize.

PATH is read by theseus-context, so it is local to the machine on which the client runs.

Examples:

./bin/theseus-context -config /path/to/config.ini new attr msg_size endo int request.messagesize
./bin/theseus-context -config /path/to/config.ini new attr gpu_util exo double
./bin/theseus-context -config /path/to/config.ini set gpu_util 0.7
./bin/theseus-context -config /path/to/config.ini new sched schedule.json 'gpu_util < 0.8' '10'

Mutation commands return accepted, node_id, and local_id after the selected daemon accepts them into its queue.

Query Commands

Queries are served by the selected daemon and are never forwarded.

Mutation Response Queries

Mutation response queries inspect retained mutation states and terminal responses.

Command	Purpose
get response LOCAL_ID	Return the retained state or terminal response for a mutation accepted by this daemon.

The query returns pending before completion, overdue when the mutation is still unfinished after mutation.response_timeout, or the mutation's terminal response after completion. overdue is not terminal and does not cancel the mutation; a later query returns its terminal response once it completes. A successful terminal response includes a secondaries map keyed by each configured secondary's node_id, with its replication result.

Response queries remain available while the daemon is synchronizing or in the inconsistent state because they only inspect retained memory.

The accepting daemon retains each accepted mutation's state and terminal response in memory, keyed by local_id, without a count or age limit, for its process lifetime. Terminal responses are retrieved with get response rather than pushed to the client after the immediate accepted response. The retained mutation-state map can grow without bound and is cleared when the daemon restarts.

Example:

./bin/theseus-context -config /path/to/config.ini get response 0123456789abcdef0123456789abcdef

Context Queries

Context queries inspect the selected daemon's local persistent context.

Command	Purpose
get context	Return the local persistent context directory as context_dir.
get attrs	List known attribute names.
get attr NAME	Return one attribute manifest and, for an exogenous attribute, the latest entry in its attribute log if one exists.
get scheds	List known schedule names.
get sched NAME	Return one schedule manifest.

Examples:

./bin/theseus-context -config /path/to/config.ini get context

Diagnostics

Diagnostic queries inspect only the local state of the daemon selected by -address, or the configured local endpoint by default.

Command	Purpose
show [status] [--output FORMAT]	Show local role, readiness, and the last applied op_index (last_applied). Bare show is an alias for show status.
show context [--details] [--output FORMAT]	Show context counts and optionally include attribute and schedule details.

FORMAT is table (the default) or json. JSON output includes schema_version, currently version 1.

show status remains available while the daemon is not_ready or inconsistent; show context requires the daemon to be ready.

Examples:

./bin/theseus-context -config /path/to/config.ini show
./bin/theseus-context -config /path/to/config.ini show status --output json
./bin/theseus-context -config /path/to/config.ini show context --details

Recovering an Inconsistent Context

If show reports state inconsistent, the daemon entered that state because a post-startup daemon session was lost, a daemon call timed out, or a persistent append had an ambiguous outcome. Context queries, show context, and mutation commands return context_inconsistent; show status and mutation response queries remain available. The context failure is persisted in <context_dir>.inconsistent and survives daemon restarts.

1. Stop or recreate the affected Theseus job.
2. Repair or reinitialize the context while Theseus is stopped.
3. Remove <context_dir>.inconsistent on every node.
4. Restart the complete configured daemon set and wait for every daemon to report ready before restarting Theseus runtimes.

Startup synchronization overwrites each secondary's persistent context, but the daemon does not terminate Theseus processes automatically. See Context Failure and Fencing for the conditions that trigger this state.

Multi-node End-to-End Test

The Docker-based E2E test starts one primary and three secondary daemon processes in separate containers. It verifies:

• independent node-local filesystems at the same configured context directory;
• synchronization over the same TCP and rsync-over-SSH paths used by the daemon;
• mutations submitted through both primary and secondary clients;
• identical persistent context files on every node; and
• all reachable daemons entering inconsistent, without reconnection, after post-startup daemon-session loss.

./tests/e2e/run-multinode.sh

Optional environment variables are:

• SECONDARY_COUNT=2 runs the same test with two secondaries.
• KEEP_CONTAINERS=1 retains the containers, network, generated configurations, and SSH keys for manual inspection.

KEEP_CONTAINERS=1 ./tests/e2e/run-multinode.sh

With KEEP_CONTAINERS=1, the script prints the retained resource names and exact cleanup commands. Otherwise, it automatically removes the containers, network, and temporary work directory; the locally built Docker image remains. The test kills and restarts one secondary container after startup; it verifies that the remaining reachable daemons enter inconsistent and that the restarted daemon does not resynchronize. It does not separately inject network or storage failures.