Mirror cron on mgrast-01: a stale lockfile silently halts all run intake, with no log or alert

[wilke]

Summary

Run intake is an rsync mirror driven by cron on mgrast-01. It is the only
automation in the entire pipeline, and it is unobservable: a stale lockfile
halts intake permanently with no alert, and all output goes to cron mail.

Current crontab (user wilke, mgrast-01)

*/2 * * * * echo hello
1 */3 * * * if [ ! -f /tmp/cron.miseq.running ] ; then touch /tmp/cron.miseq.running ; date ; time rsync -rtlP /incoming/miseq_output /nfs/seq-data/ ; rm /tmp/cron.miseq.running ; else echo Found lockfile for miseq ; fi
1 */3 * * * if [ ! -f /tmp/cron.nextseq.running ] ; then touch /tmp/cron.nextseq.running ; date ; time rsync -rtlP /incoming/nextseq2k_output /nfs/seq-data/ ; rm /tmp/cron.nextseq.running ; else echo Found lockfile for nextseq ; fi

Problems

1. A stale lockfile silently halts intake forever. After a reboot or a killed
   rsync, /tmp/cron.<x>.running persists and every subsequent tick takes the
   else branch. Nothing alerts. Runs simply stop arriving.

2. No log, no exit status. date and time output go to cron mail, which is
   almost certainly unread. "When did intake last succeed?" is currently
   unanswerable.

3. Both jobs fire in the same minute (1 */3 * * *) and compete for the same
   filer.

4. The mirror is already saturated. The miseq rsync was observed running
   3h16m against a 3h interval, so lockfile-skip is the normal steady state
   rather than an exception. Any alert must therefore key on the age of the last
   success, never on the presence of a skip.

5. */2 * * * * echo hello is cron-liveness debris.

Suggested fix (low risk, no change to the rsync itself)

Wrap the existing invocation so it appends one JSON line per run to a logfile on
/nfs/seq-data, with a three-valued outcome:

outcome	meaning	health
success	ran, rsync exit 0	healthy
skipped_locked	lockfile held, did not run	healthy — this is normal
failed	ran, non-zero exit	unhealthy

Then alert on last_success age (say >6h with no success), not on skips. Also
worth doing regardless: stagger the two jobs, and drop the echo hello line.

Not a request to replace the mirror

It works, and it is the only thing in this pipeline that runs unattended.
Instrument it before changing it.

Provenance

reports/work-260730.seqtrack.md.