A lightweight, persistent, production-inspired background job processing system built with Go and SQLite.
QueueCTL is a persistent background job processing system that uses SQLite for durable storage and supports concurrent workers, automatic retries with exponential backoff, Dead Letter Queue (DLQ) handling, scheduled jobs, priority-based processing, worker monitoring, graceful shutdown, worker crash recovery, job timeout handling, job output logging, and a read-only web dashboard.
The project is designed to demonstrate reliable background job processing with persistent state, concurrency control, failure recovery, and practical monitoring.
A complete working demonstration of QueueCTL is available below:
Watch QueueCTL Demo:
https://drive.google.com/file/d/1G2LYReHNnK-rMU5e-IX1FrvdW1TJ_5jV/view?usp=drivesdk
The demonstration covers:
- CLI overview
- Configuration
- Job submission
- Pending jobs
- Priority-based processing
- Multiple concurrent workers
- Successful job execution
- Queue status
- Automatic retries
- Explicit FAILED state
- Exponential backoff
- Dead Letter Queue
- DLQ retry
- Timeout handling
- Graceful worker shutdown
- Worker monitoring
- Worker crash recovery
- Persistence and restart behaviour
- Testing
QueueCTL provides persistent queue state and worker monitoring through its CLI.
QueueCTL provides a command-line interface for managing jobs, workers, configuration, queue status, statistics, the Dead Letter Queue, and the monitoring dashboard.
Display the complete CLI help:
go run ./cmd/queuectl --help
Jobs are submitted to the persistent SQLite-backed queue using the enqueue command and can then be viewed using the list command.
Example:
go run ./cmd/queuectl enqueue '{"id":"job1","command":"echo Hello"}'
go run ./cmd/queuectl enqueue '{"id":"job2","command":"echo QueueCTL"}'
go run ./cmd/queuectl enqueue '{"id":"job3","command":"echo Worker"}'
go run ./cmd/queuectl list
New jobs initially enter the PENDING state.
Jobs can optionally specify a numeric priority. Higher-priority jobs are selected before lower-priority jobs.
Example:
go run ./cmd/queuectl enqueue '{"id":"priority-low","command":"echo LOW","priority":1}'
go run ./cmd/queuectl enqueue '{"id":"priority-high","command":"echo HIGH","priority":10}'
Example processing order:
priority 10 → processed first
priority 1 → processed later
Jobs with equal priority retain scheduled/creation ordering.
Workers claim pending jobs, execute their commands, and transition successfully completed jobs to the COMPLETED state.
Start a worker:
go run ./cmd/queuectl worker start --count 1
View the queue:
go run ./cmd/queuectl list
View queue status:
go run ./cmd/queuectl status
When a command fails, QueueCTL records the failure using the explicit FAILED state and automatically retries the job when retry attempts remain.
With a backoff base of 2, retry delays increase approximately as:
Attempt 1 → approximately 2 seconds
Attempt 2 → approximately 4 seconds
Attempt 3 → approximately 8 seconds
The retry execution time is persisted using the job's run_at value.
Example lifecycle:
FAILED
↓
Retry scheduled
↓
PENDING
↓
PROCESSING
After the configured retry limit is exhausted, the job is moved to the Dead Letter Queue.
QueueCTL supports timeout-aware command execution using Go contexts.
Example:
go run ./cmd/queuectl enqueue '{"id":"timeout-demo","command":"sleep 35"}'
When the command exceeds the configured execution limit, it is terminated and treated as a failed execution.
The timeout failure then follows the normal retry and Dead Letter Queue workflow.
Jobs that continue to fail after exhausting their configured retry attempts are moved to the Dead Letter Queue.
.png)
View dead jobs:
go run ./cmd/queuectl dlq list
Example lifecycle:
PENDING
↓
PROCESSING
↓
FAILED
↓
Retry
↓
FAILED
↓
Retry limit exceeded
↓
DEAD
↓
DLQ
The DLQ provides a persistent location for jobs that require inspection or manual retry.
A job in the Dead Letter Queue can be manually returned to the pending queue.
Example:
go run ./cmd/queuectl dlq retry fail1
The job is returned to the PENDING state and can be picked up by a worker again.
QueueCTL supports multiple workers processing different jobs concurrently.
Start multiple workers:
go run ./cmd/queuectl worker start --count 3
Example:
worker-1 → job-A
worker-2 → job-B
worker-3 → job-C
Workers independently claim available jobs.
Atomic database transactions and state checks prevent two workers from successfully claiming the same pending job.
QueueCTL supports graceful worker shutdown using signal handling and the worker-stop mechanism.
Signal-based shutdown:
Ctrl+C
The worker receives the shutdown signal and stops accepting new work while allowing an already-claimed job to finish.
Expected shutdown behaviour:
Shutdown signal received...
Worker stopped.
All workers stopped gracefully.
Workers can also be stopped using:
go run ./cmd/queuectl worker stop
QueueCTL provides detailed worker monitoring and queue-state visibility.
View worker information:
go run ./cmd/queuectl workers
View queue status:
go run ./cmd/queuectl status
The monitoring information includes:
- Worker ID
- Worker PID
- Worker status
- Last heartbeat
- Total workers
- Pending jobs
- Processing jobs
- Completed jobs
- Failed jobs
- Dead jobs
- Total jobs
- Active workers
- Persistent SQLite-backed job storage
- Cobra-based command-line interface
- Job enqueueing
- Job listing
- Job state filtering
- Multiple concurrent workers
- Transaction-based job claiming
- Protection against duplicate job claiming
- Command execution
- Exit-code based failure detection
- Automatic retries
- Explicit FAILED state before retry/DLQ finalization
- Exponential backoff
- Configurable retry count
- Configurable backoff base
- Dead Letter Queue
- DLQ retry
- Graceful worker shutdown
- Queue status monitoring
- Worker status monitoring
- Worker PID tracking
- Worker heartbeat tracking
- Worker ownership tracking
- Worker crash recovery
- Persistent retry scheduling
- Job timeout handling
- Job output logging
- Structured logging
- Priority-based job processing
- Scheduled job execution using run_at
- SQLite WAL mode
- SQLite busy-timeout handling
- Database lock/retry handling
- Read-only web dashboard
- Unit testing
- Integration testing
- Concurrency testing
- Actual command-execution duplicate testing
- Crash-recovery testing
- Restart-persistence testing
- Graceful-shutdown testing
- Race-detector testing
+----------------------+
| Cobra CLI |
| |
| enqueue |
| list |
| worker |
| workers |
| status |
| stats |
| config |
| dlq |
| dashboard |
+----------+-----------+
|
v
+----------------------+
| Queue Store |
| SQLite |
+----------+-----------+
|
+-----------------+-----------------+
| |
v v
+-------------------+ +-------------------+
| Job Storage | | Worker Tracking |
| | | |
| Pending | | Worker ID |
| Processing | | Worker PID |
| Completed | | Status |
| Failed | | Heartbeat |
| Dead | | Claimed Job |
+---------+---------+ +-------------------+
|
v
+-------------------+
| Worker Pool |
| |
| Worker 1 |
| Worker 2 |
| Worker 3 |
+---------+---------+
|
v
+-------------------+
| Command Execution |
+---------+---------+
|
+----+----+
| |
v v
Success Failure
| |
v v
Completed FAILED
|
+-----+-----+
| |
Retry Limit reached
| |
v v
Pending DEAD
|
v
DLQ
+---------+
| PENDING |
+----+----+
|
v
+-------------+
| PROCESSING |
+------+------+
|
+-----------+-----------+
| |
Success Failure
| |
v v
+-------------+ +---------------+
| COMPLETED | | FAILED |
+-------------+ +-------+-------+
|
Retry limit exceeded?
/
No Yes
| |
v v
Exponential Backoff DEAD
| |
v v
PENDING DLQ
The explicit FAILED state is persisted during failure handling.
When retries remain:
FAILED → PENDING
When the retry limit is exhausted:
FAILED → DEAD
Start Worker
|
v
Register Worker
|
v
Update Heartbeat
|
v
Claim Next Available Job
|
+----------------------+
| |
Job Available No Job
| |
v v
Processing Worker Idle
|
v
Execute Command
|
+--+--+
| |
v v
Success Failure
| |
v v
Done FAILED
|
Retry / DLQ
QueueCTL uses SQLite as its persistent storage layer.
Job information remains available across separate CLI and worker process executions.
Persisted job information includes:
- Job ID
- Command
- State
- Attempts
- Retry limit
- Priority
- Scheduled execution time
- Retry execution time
- Execution output
- Failure information
- Worker ID
- Worker PID
- Claim timestamp
- Creation timestamp
- Update timestamp
SQLite WAL mode and busy-timeout handling improve concurrent access by multiple local workers.
QueueCTL uses database transactions to coordinate concurrent workers.
The claim process is:
1. Find an eligible pending job.
2. Respect priority.
3. Respect scheduled execution time.
4. Transition the job to PROCESSING.
5. Store worker ownership information.
6. Commit the transaction.
The atomic state transition prevents multiple workers from successfully claiming the same pending job during normal operation.
This behaviour is also verified through concurrency and stress testing.
QueueCTL supports multiple workers:
go run ./cmd/queuectl worker start --count 3
Different workers can process different jobs simultaneously:
worker-1 → parallel1
worker-2 → parallel2
worker-3 → parallel3
QueueCTL includes a stress test that submits 100 jobs and runs them using 10-worker and 20-worker configurations.
Expected result:
jobs submitted = 100
executions = 100
completed jobs = 100
duplicate executions = 0
The test validates actual command execution rather than only checking database state.
When command execution fails, QueueCTL records the failure and enters the explicit FAILED state.
If retries remain, the job is scheduled again using exponential backoff.
The retry model is:
delay = backoff_base ^ attempt
With:
backoff_base = 2
the retry delays increase approximately as:
2 seconds
4 seconds
8 seconds
...
Retry timing is persisted using run_at, allowing retry schedules to survive process restarts.
Workers do not remain blocked while waiting for a retry and can continue processing other available jobs.
When a job exhausts its retry attempts:
FAILED → DEAD
The job becomes available in the Dead Letter Queue.
List dead jobs:
go run ./cmd/queuectl dlq list
Retry a dead job:
go run ./cmd/queuectl dlq retry fail1
A DLQ retry returns the job to:
PENDING
The retried job can then be processed again by a worker.
Jobs can optionally specify a numeric priority.
Higher priority values are selected before lower priority values.
Example:
go run ./cmd/queuectl enqueue '{"id":"urgent","command":"echo urgent","priority":10}'
go run ./cmd/queuectl enqueue '{"id":"normal","command":"echo normal","priority":1}'
Expected ordering:
priority 10 → first
priority 1 → later
For equal-priority jobs, scheduled/creation ordering is preserved.
Jobs can optionally specify a run_at value specifying when they become eligible for execution.
Example:
go run ./cmd/queuectl enqueue '{"id":"future1","command":"echo Future","run_at":"2026-07-24T10:00:00Z"}'
Workers only claim scheduled jobs when their execution time has been reached.
Retry scheduling also uses persisted execution times.
QueueCTL supports timeout-aware command execution using Go contexts.
If a command exceeds its configured execution limit:
Command
↓
Timeout
↓
Process terminated
↓
FAILED
↓
Retry / DLQ
Timeout failures follow the same retry and Dead Letter Queue lifecycle as other command failures.
QueueCTL captures command output during execution and persists it with the corresponding job.
This allows execution output to remain associated with the job instead of depending only on terminal output.
Persisted output provides useful information for debugging and operational inspection.
Workers maintain persistent heartbeat information in the worker tracking table.
Worker information includes:
- Worker ID
- Process ID
- Status
- Last heartbeat
- Claimed job information
The heartbeat allows QueueCTL to detect workers that have stopped updating their state.
Every claimed job records ownership information such as:
worker_id
worker_pid
claimed_at
Workers maintain a persistent heartbeat.
If a worker stops updating its heartbeat while one of its jobs remains in PROCESSING, another worker can identify the stale worker and recover the job.
The recovery transition is:
PROCESSING
↓
Stale worker detected
↓
PENDING
↓
Available to another worker
The recovered job clears its previous worker ownership information.
QueueCTL follows an at-least-once recovery model.
If an external command produces a side effect immediately before a process crash but before completion is persisted, the command may execute again after recovery.
Workers support graceful shutdown through:
Ctrl+C
and:
go run ./cmd/queuectl worker stop
A graceful shutdown:
- Stops workers from claiming new jobs.
- Allows already-claimed work to finish.
- Updates worker state.
- Exits cleanly.
Example:
Shutdown signal received...
Worker stopped.
All workers stopped gracefully.
QueueCTL supports configuration through config.json and CLI commands.
Example configuration:
{
"database_path": "queue.db",
"default_retries": 3,
"default_workers": 1,
"backoff_base": 2,
"stale_worker_timeout_seconds": 15
}
View configuration:
go run ./cmd/queuectl config list
Set maximum retries:
go run ./cmd/queuectl config set max-retries 3
Set exponential backoff base:
go run ./cmd/queuectl config set backoff-base 2
Set stale-worker timeout:
go run ./cmd/queuectl config set stale-worker-timeout 15
Queue status:
go run ./cmd/queuectl status
Worker status:
go run ./cmd/queuectl workers
Queue statistics:
go run ./cmd/queuectl stats
The monitoring commands provide visibility into:
- Pending jobs
- Processing jobs
- Completed jobs
- Failed jobs
- Dead jobs
- Total jobs
- Active workers
- Worker status
- Worker PID
- Worker heartbeat
QueueCTL includes a minimal read-only web dashboard for lightweight queue and worker monitoring.
Start the dashboard:
go run ./cmd/queuectl dashboard --addr 127.0.0.1:8080
Open:
http://127.0.0.1:8080
The dashboard provides a visual view of queue statistics and worker monitoring information.
The dashboard is intentionally read-only and lightweight.
QueueCTL uses an explicit FAILED state during failure handling:
PENDING
↓
PROCESSING
↓
FAILED
↓
PENDING (if retries remain)
↓
PROCESSING
When the retry limit is exhausted:
FAILED
↓
DEAD
↓
DLQ
Retry timing is persisted in run_at, so scheduled retries survive process restarts.
Workers periodically update their heartbeat while idle and while executing a job.
The workers table stores:
- Worker ID
- Process ID
- Status
- Last heartbeat
The jobs table stores:
- Worker ID
- Worker PID
- Claim timestamp
Claim information is cleared when the job completes, fails, or is recovered.
QueueCTL includes a stress test that verifies actual command execution rather than only checking database state.
It submits 100 jobs and runs them with 10 and 20 concurrent workers.
Each command creates an execution marker. A second execution creates a duplicate marker and fails the test.
Expected result:
jobs submitted = 100
executions = 100
completed jobs = 100
duplicate executions = 0
This validates that concurrent workers do not incorrectly execute the same job multiple times under the tested local-worker conditions.
Pending jobs remain available after worker or CLI process restart.
Retry-scheduled jobs retain their persisted run_at value and are not claimed before that time.
Persistence is provided by SQLite, so job state does not depend solely on the lifetime of an individual worker process.
Example:
Job submitted
↓
PENDING
↓
Worker processes job
↓
Process restarts
↓
SQLite retains state
↓
Worker continues processing
Format the source code:
go fmt ./...
Run the complete test suite:
go test ./...
Run tests with the Go race detector:
go test -race ./...
Run static analysis:
go vet ./...
Check for whitespace errors:
git diff --check
The project includes tests covering:
- Queue operations
- Job claiming
- Job state transitions
- Worker behaviour
- Automatic retries
- Exponential backoff
- Timeout handling
- Dead Letter Queue
- DLQ retry
- Priority processing
- Concurrent workers
- Duplicate execution
- Worker crash recovery
- Restart persistence
- Retry persistence
- Graceful shutdown
- Worker heartbeats
- Race conditions
- Integration scenarios
QueueCTL is designed primarily for concurrent workers on a single host.
SQLite is configured with:
- WAL mode
- Busy timeout
- Transaction-based job claiming
- Database lock/retry handling
The queue store uses atomic state transitions when claiming jobs to prevent multiple workers from processing the same pending job.
Distributed multi-host worker coordination is outside the scope of this implementation.
go run ./cmd/queuectl --help
go run ./cmd/queuectl enqueue '{"id":"job1","command":"echo Hello"}'
go run ./cmd/queuectl list
go run ./cmd/queuectl list --state pending
go run ./cmd/queuectl list --state processing
go run ./cmd/queuectl worker start --count 1
go run ./cmd/queuectl worker start --count 3
go run ./cmd/queuectl worker stop
go run ./cmd/queuectl status
go run ./cmd/queuectl workers
go run ./cmd/queuectl stats
go run ./cmd/queuectl dlq list
go run ./cmd/queuectl dlq retry job1
go run ./cmd/queuectl config list
go run ./cmd/queuectl config set max-retries 3
go run ./cmd/queuectl config set backoff-base 2
go run ./cmd/queuectl config set stale-worker-timeout 15
go run ./cmd/queuectl dashboard --addr 127.0.0.1:8080
queuectl/
├── cmd/
│ └── queuectl/
│ └── main.go
│
├── internal/
│ ├── cli/
│ │ ├── config.go
│ │ ├── dashboard.go
│ │ ├── dlq.go
│ │ ├── status.go
│ │ ├── worker.go
│ │ └── workers.go
│ │
│ ├── config/
│ │
│ ├── queue/
│ │
│ └── worker/
│
├── pkg/
│ └── logger/
│
├── images/
│ ├── QueueCTL Initial Status.png
│ ├── CLI Commands & Help.png
│ ├── Job Enqueueing & Job Listing.png
│ ├── Priority-Based Job Processing.png
│ ├── Successful Job Execution & Queue Status.png
│ ├── Automatic Retry & Exponential Backoff.png
│ ├── Job Timeout Handling.png
│ ├── Dead Letter Queue (DLQ).png
│ ├── DLQ Job Retry.png
│ ├── Multiple Concurrent Workers.png
│ ├── Graceful Worker Shutdown.png
│ └── Worker Monitoring & Final Queue Status.png
│
├── config.json
├── go.mod
├── go.sum
├── ARCHITECTURE.md
├── README.md
└── .gitignore
- Go
- Git
- macOS, Linux, or another supported Go environment
git clone https://github.com/jeevangowda2005/queuectl.git
cd queuectl
go mod tidy
go test ./...
go test -race ./...
go vet ./...
- Go — application logic and concurrency
- Cobra — command-line interface
- SQLite — persistent storage
- modernc.org/sqlite — SQLite driver
- Go Context — timeout and cancellation handling
- Go Testing — automated testing
- Go Race Detector — concurrency verification
- Go HTTP — lightweight dashboard
QueueCTL uses SQLite because it provides lightweight persistent storage without requiring an external database server.
Atomic database transactions are used to coordinate local workers.
Persistent retry scheduling avoids keeping retry timers only in memory.
Worker heartbeats allow the system to identify stale workers.
Worker ownership metadata allows processing jobs to be recovered after worker failure.
The project follows an at-least-once recovery model because external command side effects cannot automatically be rolled back after a process crash.
The dashboard is intentionally read-only and minimal.
- SQLite provides persistent storage suitable for a lightweight single-host queue.
- Atomic database transactions coordinate local workers.
- Commands are executed on the host system.
- The CLI and dashboard should not be exposed to untrusted users without appropriate authorization.
- Retry attempts and retry scheduling are persisted.
- Worker stop requests allow already-claimed work to finish.
- Worker heartbeat information is used for monitoring and crash recovery.
- Distributed multi-host coordination is outside the scope of this implementation.
- Crash recovery follows at-least-once semantics.
- External command side effects cannot be automatically rolled back.
- The dashboard is intentionally read-only and lightweight.
QueueCTL is designed around durable state transitions.
Normal successful execution:
PENDING
↓
PROCESSING
↓
COMPLETED
Failed execution with retries:
PENDING
↓
PROCESSING
↓
FAILED
↓
PENDING
↓
PROCESSING
↓
COMPLETED
Failed execution with exhausted retries:
PENDING
↓
PROCESSING
↓
FAILED
↓
DEAD
↓
DLQ
Worker crash recovery:
PROCESSING
↓
Worker heartbeat becomes stale
↓
Stale worker detected
↓
Job recovered
↓
PENDING
↓
Another worker claims the job
This architecture keeps queue state persistent and recoverable across worker lifecycle events.
Possible future enhancements include:
- Job cancellation
- REST API
- Authentication and authorization
- Distributed worker support
- Prometheus metrics
- Grafana integration
- Advanced job scheduling
- Job dependencies
- Web-based job management
- Authentication for monitoring endpoints
- Improved distributed coordination
Jeevan Gowda
GitHub:
https://github.com/jeevangowda2005/queuectl
QueueCTL demonstrates how a reliable background job processing system can be built using Go and SQLite while addressing practical concurrency, persistence, and failure-recovery requirements.
The system provides:
- Persistent job storage
- Concurrent workers
- Atomic job claiming
- Priority scheduling
- Scheduled execution
- Automatic retries
- Exponential backoff
- Explicit FAILED state
- Dead Letter Queue
- DLQ retry
- Timeout handling
- Worker heartbeats
- Worker crash recovery
- Graceful shutdown
- Job output logging
- Queue monitoring
- Web dashboard
- Concurrency stress testing
- Duplicate-execution testing
- Restart-persistence testing
- Race detection
QueueCTL is designed as a lightweight but robust background job processing system with a strong focus on persistence, concurrency safety, reliability, and operational visibility.