cpp-threadpool

A thread pool with work-stealing and dependency-graph scheduling, written in C++23. A personal project to consolidate concurrency fundamentals — mutexes, condition variables, futures/promises, and the concurrency-safety habits (ASan/UBSan/TSan, deliberate lock scoping) that come with writing multithreaded code correctly rather than just getting it to pass once.

Features

• Arbitrary task submission — submit(fn, args...) accepts any callable and argument list, returning a TaskHandle<T> (a thin wrapper over std::shared_future<T>) via std::invoke_result_t and variadic templates. Return type and exceptions both propagate through to .get().
• Dependency graph scheduling — submit_with_deps(deps, fn, args...) lets a task wait on the completion of others before it becomes eligible to run. Dependents are tracked via a waiter_list on each task rather than a central registry; a task becomes runnable the moment its dependency count reaches zero.
• Failure propagation — if a dependency throws, everything downstream of it (including transitively, through multi-level chains and diamond-shaped graphs) is skipped rather than run, and the original exception surfaces at .get() on any dependent — not just the task that actually failed.
• Work-stealing — each worker owns its own deque. Newly-ready dependent tasks are pushed onto the worker that just completed their dependency (for cache locality), and idle workers steal from each other's deques when their own is empty rather than blocking on a single shared queue.
• Clean shutdown — the pool signals every worker to stop and joins all of them before any worker's internal state is torn down, avoiding a use-after-free between still-running workers and the ones already being destroyed.

Architecture

flowchart LR
    submit["submit() / submit_with_deps()"]

    subgraph W0["Worker 0"]
        Q0[["deque"]]
    end
    subgraph W1["Worker 1"]
        Q1[["deque"]]
    end
    subgraph W2["Worker 2"]
        Q2[["deque"]]
    end

    submit -->|round-robin| Q0
    submit -->|round-robin| Q1
    submit -->|round-robin| Q2

    Q0 -.->|steal when idle| Q2
    Q2 -.->|steal when idle| Q0

Loading

Fresh submissions are spread round-robin across workers. When a worker finishes a task, any dependent tasks that just became ready are pushed onto that same worker's own queue (cache-local, per the design notes above) rather than round-robined — which is also why a worker can end up with more queued work than its neighbors. Idle workers correct that imbalance by stealing from a busy neighbor's queue rather than blocking.

Design notes

• PIMPL on both ThreadPool and Worker, keeping <mutex>, <condition_variable>, <deque>, and <thread> out of the public headers.
• Type erasure via std::function wraps each task's std::packaged_task<R()> so the pool's internal queues can hold tasks of any return type uniformly.
• std::jthread + std::stop_token for worker lifecycle — cooperative shutdown without a hand-rolled atomic flag.
• Each Worker owns its own deque and its own mutex — deliberately fine-grained, since this is the point of the per-worker-queue design: one worker's queue operations never serialize against another's.
• Dependency-graph bookkeeping (waiter_list, dep_count, completed/failed) is protected by one mutex shared across the whole pool, not per-task. A submit_with_deps for task A and a complete_task for an unrelated task B will serialize against each other under this scheme, even though they touch disjoint data — a per-task mutex would remove that, at the cost of real complexity. Chosen as the simpler default on the assumption that task granularity (loops, sleeps, string formatting) dwarfs the time spent holding this lock — a named contention/simplicity tradeoff, not an oversight.

Known limitations

• No cycle detection. A cyclic dependency graph will deadlock rather than fail fast at submission time.
• No central pending-task registry. Tasks blocked on a dependency are reachable only via their dependency's waiter_list; there's no O(1) view of everything currently pending, and tasks still pending at pool destruction are silently dropped rather than erroring.

Both are accepted scope boundaries for a learning project, not oversights — a production scheduler would need at least the first.

Build & test

cmake -S . -B build
cmake --build build -j$(nproc)
ctest --test-dir build --output-on-failure

Sanitizer builds use separate build directories, since ASan/UBSan and TSan can't coexist in one binary:

# ASan + UBSan (default)
cmake -S . -B build-asan
cmake --build build-asan -j$(nproc)
ctest --test-dir build-asan --output-on-failure

# ThreadSanitizer
cmake -S . -B build-tsan -DENABLE_TSAN=ON
cmake --build build-tsan -j$(nproc)
ctest --test-dir build-tsan --output-on-failure

CI runs three jobs on every push and pull request: actionlint (lints the workflow file itself), then the ASan+UBSan and TSan builds/test suites in parallel.

To run the same workflow lint locally before pushing:

curl -sL https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz | tar xz
sudo mv actionlint /usr/local/bin/
actionlint

No output and exit code 0 means clean. actionlint bundles shellcheck for the run: blocks, so it catches shell-scripting issues (like unquoted command substitutions) in addition to workflow YAML errors.

Project structure

include/          Public headers (Task.h, ThreadPool.h, Worker.h)
src/               Implementation (ThreadPool.cpp, Worker.cpp, main.cpp)
tests/             GoogleTest suite (fetched via FetchContent)
.github/workflows/ CI (workflow lint, ASan+UBSan, and TSan, run as separate jobs)

Requirements

• C++23 compiler (developed against GCC 14)
• CMake ≥ 3.20