fix(spurctld): report BeginTime while --begin defers a job

[maybeharshit]

Motivation

A job held back by --begin sat in PENDING with Reason=None — the same string shown for a job the scheduler has simply not reached yet. Nothing distinguished a deliberate deferral from an idle scheduling pass, on squeue, scontrol show job, or the REST state_reason field. Slurm reports BeginTime for this case.

Fixes #547.

Technical Details

classify_pending_jobs marked a begin-deferred job scheduling_eligible = false and then dropped it at the closing candidates.retain(...) without ever pushing it onto the blocked list that feeds apply_blocked_pending_reasons. It reached none of the blocker checks either, since those all pass ineligible candidates straight through, so no code path could name its wait. PendingReason::BeginTime was therefore only ever set by the preemption-requeue path.

Notably, the filter at the top of the same function already short-circuits jobs that are begin-held and already tagged BeginTime, so a tagger was anticipated — it was just never written.

The hold is now recorded at that final retain, keyed on the existing Job::is_begin_held predicate rather than on !scheduling_eligible, so only a genuine begin-time deferral is tagged.

Two consequences of where it sits:

[1] Placing it after the blocker checks preserves their existing precedence. A deferred job that also sits in a down partition still reports PartitionInactive, which explains why it cannot run at all, rather than naming a start time it will never reach. There is a test for this.

[2] Consumable shortages (licenses, burst buffer) continue to be skipped for deferred jobs. Those are transient and may well be resolved by the time the start time arrives, so the hold is the honest reason to show.

The scheduler loop already calls pending_jobs_and_tag_reasons() before its pending.is_empty() early return, specifically so reasons stay fresh when nothing is schedulable — which is exactly the issue's single-deferred-job-on-an-idle-cluster scenario, so no change was needed there.

Note on a changed test expectation

future_begin_jobs_do_not_receive_consumable_block_reasons asserted that a deferred job's reason was left untouched, using that as a proxy for "no consumable shortage was recorded". That invariant still holds — the license and burst-buffer checks continue to skip ineligible candidates — but the proxy no longer works, since the reason is now the deferral instead of whatever the job carried before. The assertions were updated to name the real invariant.

Test Plan

• Unit tests in spurctld for the deferral being tagged, the reason giving way once the hold lapses, and a structural blocker outranking the hold. The first two were confirmed to fail without the fix.
• New tests/native_host/e2e/test_begin_time.py covering the reported flow plus a control job with no --begin.
• Clippy, fmt, and the full workspace suite.
• Live single-node LXD cluster (Ubuntu 24.04) reproducing the issue's commands.

Test Result

Clippy and fmt clean; full workspace suite passes. Both new unit tests fail on main with left: None, right: BeginTime, matching the reported symptom exactly.

On the LXD cluster, sbatch --begin=now+40seconds ok.sh:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
                 1   default /root/ok     root PD       0:00      1 (BeginTime)

$ scontrol show job 1
   JobState=PENDING Reason=BeginTime

The REST surface named in the issue agrees — GET /slurm/v0.0.42/job/2 returned job_state = PENDING with state_reason = BeginTime.

The deferral itself still works: the job started 41 seconds after submit for a 40-second --begin, ran to completion, and its reason did not remain stuck at BeginTime.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

Made with Cursor

[yansun1996]

Hi @maybeharshit , you marked this PR as draft, are you still working on more changes ?

[shiv-tyagi]

Posted comments. PTAL.

[shiv-tyagi]

Nit: this is slightly verbose for what it communicates. The first sentence is the insight; the rest restates what's visible from the surrounding code (the blocker checks all return true for ineligible candidates).

Suggested change

	// A begin-deferred job reaches none of the blocker checks above, which
	// all pass ineligible candidates straight through, so this is the only
	// place its wait can be named. Structural blockers already claimed the
	// jobs they explain, keeping their precedence over the hold.
	// This is the only tagger for begin-deferred jobs: the blocker checks
	// above pass ineligible candidates through without claiming them.