feat(spurctld): add job submission validation hook (shell + lua)

[yansun1996]

Summary

Adds a site-controlled job-submission hook — a Spur equivalent of Slurm's job_submit plugin interface — with two backends:

• Shell (hooks.job_submit): a script gets the resolved spec as JSON on stdin and decides via exit code + stdout (Spur-native).
• Lua (hooks.job_submit_lua): a sandboxed script defines slurm_job_submit(job_desc, submit_uid) and mutates job_desc in place — literal parity with Slurm's job_submit/lua, so an existing Slurm Lua policy ports with minimal changes.

Both run on the controller at submission (on the leader, before the job enters the Raft log) and can accept, reject (message shown to the user), or modify the job. If both are configured, shell runs first, then Lua, each on the evolving spec (mirroring Slurm's config-ordered plugin chain). Example policies: examples/hooks/job_submit.sh, examples/hooks/job_submit.lua.

The eight items in the request (check user/group/partition/GPUs, enforce walltime, auto-add QoS, add constraints, reject bad jobs, audit) are all expressible as site policy in either backend.

Approach

• Whitelisted modify. Both backends may only change policy/scheduling fields: qos, partition, account, constraint, comment, reservation, priority, time_limit (minutes), begin_time, gres, hold. Changes are routed through one typed parser, so identity (user/uid/gid), the script/argv, and resource-count fields are structurally unmodifiable — a policy script cannot forge identity or alter what runs. Non-whitelisted keys are ignored + logged (shell) / never read (Lua).
• Lua sandbox. The interpreter runs with only table/string/math/utf8/coroutine; os, io, package/require, and debug are excluded, and the always-loaded base globals that reach the filesystem or bytecode loader (dofile, loadfile, load, loadstring, collectgarbage) are removed. A memory ceiling and an instruction-count interrupt guard against a runaway policy hanging or OOMing the controller. time_limit is exposed and accepted in minutes (Slurm convention); unset fields read as Lua nil.
• Hook point. Runs in submit_job after default/QoS/account resolution and validation, before the size check and Raft propose, so edits are what get persisted/scheduled.
• Fail-closed. A shell hook emitting unparseable JSON, a wrong-typed field, a Lua syntax/runtime error, a missing slurm_job_submit, or an exceeded resource limit all reject the submission rather than silently accepting it.
• Post-modify re-validation. A hook-changed partition/account is re-checked against ACLs; a hook-set QoS is trusted (not re-authorized) but still existence-checked so an unknown QoS can't silently resolve to the limitless default.
• Audit logging. Each decision (accept / reject / modified fields) is logged under a stable audit target with user/uid/partition/gpu context, tagged per backend (job_submit / job_submit_lua).
• Upgrade-safe. The new config fields are additive/optional and not part of any persisted Raft/WAL type; proto is unchanged. Lua is vendored (Lua 5.4 built from source) so there is no system-Lua build dependency.

Known limitations

• If a hook sets gres GPU entries alongside a user's explicit --gpus, the requests can conflict; since GPU demand is resolved later, this surfaces at schedule time, not submit.
• The shell example requires jq; if absent it fails closed (all submissions rejected) — noted in the script header.

Test plan

• Unit tests for both backends: accept / reject (message surfaced) / modify / non-whitelisted-field-ignored / every whitelisted field / fail-closed (malformed shell JSON, wrong type, Lua syntax/runtime error, missing entry point). Lua adds: sandbox denies os/io/require/dofile/loadfile/load, infinite loop is interrupted, time_limit minutes + whole-valued float, unchanged fields not reported, unset field reads as nil. Lua also audits non-whitelisted fields a script set (ignored-key parity with shell).
• Controller wiring tests: shell and Lua reject/modify; modify to invalid partition and unknown QoS both rejected; shell→Lua chain; shell reject short-circuits Lua; Lua overrides a shell-set field.
• Native-host e2e (tests/native_host/e2e/test_job_submit_hook.py) for both backends: reject reaches the CLI, modify persists and is queryable via scontrol, unconfigured hook is inert.
• cargo clippy --workspace --exclude spur-ffi --all-targets clean; cargo fmt --all --check clean; targeted spur-core/spurctld suites pass.
• Validated end-to-end on an isolated deployment (both backends): reject surfaces the script's message; auto-QoS / priority / comment / walltime-cap modifies persist and show in scontrol; unknown-QoS rejected; unconfigured hook inert; shell malformed-JSON fails closed. Lua specifically: os.execute and dofile fail closed with no filesystem effect, an infinite loop is interrupted in well under a second while the controller stays responsive, and the shell→Lua chain applies both edits with distinct audit lines.

Design notes

• Hook-set QoS is trusted policy. A QoS set by the hook is checked for existence but not re-run through the per-user allow-list ACL (unlike a user-supplied --qos). The hook is the policy authority, so it can grant a QoS the user could not request directly.
• time_limit encoding differs by backend. The shell hook receives the resolved spec verbatim, where time_limit is a [seconds, nanos] array; the Lua hook receives it as integer minutes (Slurm job_submit.lua convention). Both example scripts show the correct handling.
• Over-cap time_limit pends, not rejects. A hook may raise time_limit above a partition's max; like an over-cap user submission, the job stays pending (the partition max-time check is a schedule-time gate), rather than being rejected at submit.

Addresses SPUR-62.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 97.28643% with 27 lines in your changes missing coverage. Please review.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #565      +/-   ##
==========================================
+ Coverage   75.83%   76.16%   +0.33%     
==========================================
  Files         166      166              
  Lines       64696    65691     +995     
==========================================
+ Hits        49058    50031     +973     
- Misses      15638    15660      +22     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Adds a site-configurable job submission hook to spurctld (implemented as an external script) so clusters can enforce and/or mutate policy at submit-time before the job is persisted into the Raft log.

Changes:

• Wire hooks.job_submit into controller submission flow, applying a whitelisted set of spec edits and surfacing hook rejections to the caller.
• Introduce a core hook runner and change-application helpers for the submit hook (stdin JSON contract, stdout change parsing).
• Add documentation/examples plus unit + native-host e2e coverage for accept/reject/modify behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/native_host/e2e/test_job_submit_hook.py	New native-host e2e coverage for reject/modify/inert behavior through the CLI→controller path.
examples/spur.conf	Documents the new optional [hooks].job_submit configuration.
examples/hooks/job_submit.sh	Provides an example policy hook script demonstrating reject + modify patterns.
crates/spurctld/src/cluster.rs	Runs the submit hook during submit_job and applies hook-directed changes.
crates/spur-core/src/hooks.rs	Implements submit-hook execution, stdout parsing into typed changes, and spec mutation helpers.
crates/spur-core/src/config.rs	Adds HooksConfig.job_submit as an additive optional config field.

Suppressed comments (2)

crates/spur-core/src/hooks.rs:257

• time_limit_minutes currently accepts negative (and extremely large) integers. A negative value will create a negative chrono::Duration later, which can bypass partition max-time checks (tl.num_minutes() > max) and lead to nonsensical scheduling behavior. Consider validating this field as a non-negative, non-overflowing minute count and failing closed when invalid.

            "reservation" => changes.reservation = Some(take_string(key, value)?),
            "priority" => changes.priority = Some(take_u32(key, value)?),
            "time_limit_minutes" => changes.time_limit_minutes = Some(take_i64(key, value)?),
            "begin_time" => changes.begin_time = Some(take_datetime(key, value)?),

crates/spur-core/src/hooks.rs:244

• The context string "job_submit hook emitted unparseable JSON" is misleading when stdout is valid JSON but not an object (e.g., a number/array). Since serde_json::from_str here expects an object/map, the error should reflect the expected shape to make hook misconfigurations easier to diagnose.

fn parse_submit_changes(stdout: &str) -> anyhow::Result<SubmitHookChanges> {
    let map: serde_json::Map<String, serde_json::Value> =
        serde_json::from_str(stdout).context("job_submit hook emitted unparseable JSON")?;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

crates/spur-core/src/hooks.rs:346

• LUA_INSTRUCTION_LIMIT is set to 100,000,000 instructions. Even though this bounds infinite loops, it can still translate to multi-second CPU stalls per submission on slower hardware, and submit_job runs synchronously on the async gRPC worker thread. Consider lowering the instruction budget (or additionally enforcing a wall-clock timeout) to keep a misbehaving policy from degrading controller availability.

/// Instruction budget before a lua script is interrupted (guards infinite loops).
const LUA_INSTRUCTION_LIMIT: u32 = 100_000_000;

crates/spur-core/src/hooks.rs:315

• job_submit changes allow time_limit_minutes to be negative or extremely large. A negative value can bypass later max_time_minutes checks (because tl.num_minutes() becomes negative), and a very large value risks overflow when it is converted into a chrono::Duration. Since hooks are expected to be fail-closed on bad input, validate time_limit_minutes during parsing (both shell + Lua paths share take_i64).

fn take_i64(key: &str, value: &serde_json::Value) -> anyhow::Result<i64> {
    // Lua arithmetic yields floats (`x / 2`), so accept a whole-valued float too.
    if let Some(f) = value.as_f64() {
        if value.as_i64().is_none() && f.fract() == 0.0 && f.is_finite() {
            return Ok(f as i64);

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

crates/spur-core/src/hooks.rs:254

• On timeout the hook process is killed but never awaited/reaped. On Unix this can leave a zombie process behind until spurctld exits, and repeated timeouts could exhaust the process table. After killing the child, wait for it (best-effort, with a short timeout) before bailing.

        Err(_) => {
            let _ = child.kill().await;
            anyhow::bail!(
                "job_submit hook timed out after {SUBMIT_HOOK_TIMEOUT_SECS}s (script: {script_path})"
            );
        }

crates/spurctld/src/cluster.rs:408

• run_submit_hook_lua does file IO + potentially heavy CPU work (Lua execution/instruction budget) but is called directly from submit_job, which is typically executed on a Tokio worker thread via the gRPC/REST handlers. Unlike the shell path, this can block the runtime and reduce controller throughput. Consider running the Lua hook inside block_in_place (or spawn_blocking) as well, matching the shell hook’s scheduling behavior.

        if let Some(script) = lua {
            let ctx = self.submit_hook_ctx(spec)?;
            let outcome = spur_core::hooks::run_submit_hook_lua(script, &ctx)
                .map_err(|e| SubmitError::internal(format!("job_submit lua hook failed: {e}")))?;
            self.apply_submit_outcome(spec, "job_submit_lua", outcome)?;