QueueCTL

A persistent background job queue, built for the terminal.

QueueCTL is a CLI-based persistent background job queue written in Go. It supports job enqueueing, foreground workers, retry with exponential backoff, dead-letter handling, crash recovery, and configuration through a command-line interface — all backed by an embedded SQLite store, so there's no external broker to stand up.

Demo

Click to watch setup, CLI usage, and crash recovery in action.

---

Why QueueCTL

• Zero infrastructure — SQLite is embedded via modernc.org/sqlite; no broker, no server, no C compiler.
• Crash-safe by design — a stale-job sweeper and worker heartbeats recover jobs left behind by a killed worker.
• Predictable retries — exponential backoff with a configurable base, and a dead-letter queue for jobs that exhaust their attempts.
• Concurrency without ceremony — a single atomic UPDATE ... RETURNING statement means multiple workers can never claim the same job.

---

Requirements

• Go 1.26 or newer
• Windows PowerShell, Bash, or any shell that can run Go commands
• SQLite is embedded through modernc.org/sqlite; no external SQLite server or C compiler is required

Setup

git clone https://github.com/ANAS727189/queueCTL
cd queuectl
go mod download
go build -o queuectl .

During local development, you can also run the CLI without building:

go run . --help

The app stores data in queuectl.db in the current working directory.

---

CLI Usage

Enqueue a job:

./queuectl enqueue '{"id":"job1","command":"echo hello"}'

PowerShell users can use stop-parsing mode when passing JSON:

go run . --% enqueue "{\"id\":\"job1\",\"command\":\"echo hello\"}"

Start workers in the foreground:

./queuectl worker start --count 3

Stop workers gracefully from another terminal:

./queuectl worker stop

Show queue state and active worker count:

./queuectl status

List jobs by state:

./queuectl list --state pending
./queuectl list --state completed --json

Inspect and retry the dead-letter queue:

./queuectl dlq list
./queuectl dlq retry job1

Configure retry behavior:

./queuectl config set max-retries 3
./queuectl config set backoff-base 2

---

Architecture

QueueCTL uses SQLite as the persistent store. Jobs live in a jobs table with the states pending, processing, completed, failed, and dead. Configuration is stored in a config table, and foreground worker processes register in a workers table so another terminal can request graceful shutdown.

State	Meaning
pending	Queued, waiting for a worker to claim it
processing	Claimed by a worker, currently executing
completed	Finished with a zero exit code
failed	Non-zero exit, scheduled for retry
dead	Exhausted max-retries, sitting in the DLQ

Workers repeatedly claim the next eligible job, execute its command through the OS shell, and update the final state based on the exit code. Failed jobs are rescheduled with exponential backoff:

delay = backoff-base ^ attempts seconds

After max_retries failed attempts, the job moves to the DLQ with state dead.

Concurrency safety comes from a single atomic SQLite UPDATE ... RETURNING statement in internal/db/jobs.go. SQLite serializes writers across processes, so two workers cannot claim the same row at the same time.

Crash recovery is handled by a stale-job sweeper plus worker heartbeats. If a worker dies while a job is processing, a later worker reclaims jobs whose updated_at timestamp is older than 30 seconds and whose owning worker heartbeat is no longer fresh. The sweeper runs every 5 seconds, so default worst-case recovery is under 35 seconds once workers are running again.

More detailed trade-offs are documented in DECISIONS.md and design.md.

---

Testing

Run package checks:

go test ./...

Run the smoke test:

python3 scripts/smoke_test.py

On Windows, use python scripts\smoke_test.py if your Python launcher is named python.

The smoke test builds the CLI, uses an isolated temporary database, and verifies:

• a basic job completes
• a failing job retries and reaches the DLQ
• multiple workers process multiple jobs
• a forcibly killed worker leaves a job recoverable by a later worker

---

Assumptions and Trade-offs

• Job commands are trusted shell commands supplied by the CLI user.
• Worker shutdown is cooperative: worker stop, Ctrl+C, or SIGTERM lets in-flight jobs finish before exit.
• SIGKILL cannot run cleanup code, so crash recovery uses a timeout plus worker-heartbeat check.
• max-retries is copied into a job when it is enqueued. Changing it later affects new jobs, not existing jobs.
• backoff-base is read when a retry is scheduled, so changes can affect already-enqueued failed jobs.