Skip to content

perf: reduce amount of used memory and threads #7659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
knst wants to merge 4 commits into
base: develop
Choose a base branch
from
Open

perf: reduce amount of used memory and threads #7659

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f36e5e2
test: run test nodes with -par=2 -parbls=2 -rpcthreads=2 by default
knst Sep 5, 2026
a739cf7
test: run test nodes with -maxsigcachesize=1 by default
knst Sep 5, 2026
756fc4f
perf: warm quorum public key shares on the BLS worker pool
knst Sep 6, 2026
e4f459d
perf: open the masternode connection thread on demand
knst Sep 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions src/bls/bls_worker.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,11 @@ void CBLSWorker::Stop()
workerPool.stop(true);
}

void CBLSWorker::PushJob(std::function<void()> job)
{
workerPool.push([job = std::move(job)](int) { job(); });
}

#ifndef BUILD_BITCOIN_INTERNAL
bool CBLSWorker::GenerateContributions(int quorumThreshold, Span<CBLSId> ids, BLSVerificationVectorPtr& vvecRet, std::vector<CBLSSecretKey>& skSharesRet)
{
Expand Down
1 change: 1 addition & 0 deletions src/bls/bls_worker.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ class CBLSWorker

void Start(int16_t worker_count);
void Stop();
void PushJob(std::function<void()> job);

#ifndef BUILD_BITCOIN_INTERNAL
bool GenerateContributions(int threshold, Span<CBLSId> ids, BLSVerificationVectorPtr& vvecRet, std::vector<CBLSSecretKey>& skSharesRet);
Expand Down
3 changes: 3 additions & 0 deletions src/init.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2562,6 +2562,9 @@ bool AppInitMain(NodeContext& node, interfaces::BlockAndHeaderTipInfo* tip_info)
connOptions.m_peer_connect_timeout = peer_connect_timeout;
connOptions.socketEventsMode = ::g_socket_events_mode;
connOptions.m_active_masternode = node.active_ctx != nullptr;
// wallets may mix with CoinJoin, which connects to the mixing masternode
connOptions.m_masternode_connections = node.active_ctx != nullptr || quorums_watch ||
node.wallet_loader != nullptr;

// Port to bind to if `-bind=addr` is provided without a `:port` suffix.
const uint16_t default_bind_port =
Expand Down
1 change: 1 addition & 0 deletions src/llmq/context.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,5 +28,6 @@ LLMQContext::LLMQContext(CDeterministicMNManager& dmnman, CEvoDB& evo_db, Chains

LLMQContext::~LLMQContext()
{
qman->InterruptWarming();
bls_worker->Stop();
}
54 changes: 12 additions & 42 deletions src/llmq/quorumsman.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@
#include <chainparams.h>
#include <dbwrapper.h>
#include <logging.h>
#include <util/thread.h>
#include <util/time.h>
#include <validation.h>

Expand All @@ -40,18 +39,10 @@ CQuorumManager::CQuorumManager(CBLSWorker& _blsWorker, CDeterministicMNManager&
db{util::MakeDbWrapper({db_params.path / "llmq" / "quorumdb", db_params.memory, db_params.wipe, /*cache_size=*/1 << 20})}
{
mapQuorumsCache.Init(m_chainman.GetConsensus(), /*limit_by_connections=*/false);
m_cache_interrupt.reset();
m_cache_thread = std::thread(&util::TraceThread, "q-cache", [this] { CacheWarmingThreadMain(); });
MigrateOldQuorumDB(_evoDb);
}

CQuorumManager::~CQuorumManager()
{
if (m_cache_thread.joinable()) {
m_cache_interrupt();
m_cache_thread.join();
}
}
CQuorumManager::~CQuorumManager() = default;

bool CQuorumManager::GetEncryptedContributions(Consensus::LLMQType llmq_type, const CBlockIndex* block_index,
const std::vector<bool>& valid_members, const uint256& protx_hash,
Expand Down Expand Up @@ -505,50 +496,29 @@ void CQuorumManager::WriteContributions(const CQuorumPtr& quorum) const
quorum->WriteContributions(*db);
}

void CQuorumManager::CacheWarmingThreadMain() const
void CQuorumManager::QueueQuorumForWarming(CQuorumCPtr pQuorum) const
{
while (!m_cache_interrupt) {
CQuorumCPtr pQuorum;
{
LOCK(m_cache_cs);
if (!m_cache_queue.empty()) {
pQuorum = std::move(m_cache_queue.front());
m_cache_queue.pop_front();
};
}

if (!pQuorum) {
m_cache_interrupt.sleep_for(std::chrono::milliseconds(100));
continue;
}

if (!pQuorum->HasVerificationVector()) {
return;
}
// The job may reference this manager: ~LLMQContext stops the worker pool before destroying it.
blsWorker.PushJob([this, pQuorum = std::move(pQuorum)]() {

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep cache warming from starving BLS verification

When a watcher loads many verification vectors, each quorum adds a long-running warming job to the same FIFO CBLSWorker pool used by AsyncVerifySig; scans can populate up to 64 quorums, so signature, ChainLock, and InstantSend verification submitted afterward cannot run until all earlier warmers finish. The former dedicated thread kept warming off the latency-sensitive verification pool; retain a single outstanding warming task or otherwise prioritize verification work.

AGENTS.md reference: AGENTS.md:L211-L212

Useful? React with 👍 / 👎.

cxxtimer::Timer t(true);
LogPrint(BCLog::LLMQ, "CQuorumManager::%s -- type=%d height=%d hash=%s start\n", __func__,
LogPrint(BCLog::LLMQ, "CQuorumManager::QueueQuorumForWarming -- type=%d height=%d hash=%s start\n",
std23::to_underlying(pQuorum->params.type), pQuorum->m_quorum_base_block_index->nHeight,
pQuorum->m_quorum_base_block_index->GetBlockHash().ToString());

// when then later some other thread tries to get keys, it will be much faster
for (const auto i : util::irange(pQuorum->members.size())) {
if (m_cache_interrupt) {
break;
if (m_warming_interrupted) {
return;
}
if (pQuorum->qc->validMembers[i]) {
pQuorum->GetPubKeyShare(i);
Comment on lines 510 to 515

@thepastaclaw thepastaclaw Sep 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Let required BLS work run between bounded warming batches

This loop occupies one BLS worker until every valid member's public-key share has been computed. DEFAULT_WORKER_COUNT is one on a two-logical-CPU host, and the pool uses a FIFO queue with no priority or interleaving within a job. NetQuorum::ProcessContribQDATA() calls AggregateSecretKeys() synchronously at net_quorum.cpp:345; that function submits aggregation to this same pool and waits on its future. Consequently, warming another quorum blocks recovery and the P2P message-processing thread even when the recovering quorum already has its verification vector. This affects ordinary masternodes, not just -watchquorums, and a 400-member quorum with a 340-coefficient verification vector entails substantial share-computation work before the worker becomes available. Additional queued warmers extend the delay. The former dedicated warming thread did not impose this queue dependency. Split warming into small resumable batches and bound outstanding warming work so required aggregation and DKG jobs can interleave, with a single-worker regression test.

source: ['claude']

@thepastaclaw thepastaclaw Sep 9, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Withdrawn (re-reviewed at e4f459d9): I am withdrawing this duplicate blocking item and retaining the shared-pool scheduling concern once, as a non-blocking suggestion. Revalidation confirms added FIFO contention, but does not establish a correctness or protocol-deadline failure warranting a blocker.

}
}

LogPrint(BCLog::LLMQ, "CQuorumManager::%s -- type=%d height=%d hash=%s done. time=%d\n", __func__,
LogPrint(BCLog::LLMQ, "CQuorumManager::QueueQuorumForWarming -- type=%d height=%d hash=%s done. time=%d\n",
std23::to_underlying(pQuorum->params.type), pQuorum->m_quorum_base_block_index->nHeight,
pQuorum->m_quorum_base_block_index->GetBlockHash().ToString(), t.count());
}
}

void CQuorumManager::QueueQuorumForWarming(CQuorumCPtr pQuorum) const
{
if (pQuorum->HasVerificationVector()) {
LOCK(m_cache_cs);
m_cache_queue.push_back(std::move(pQuorum));
}
});
Comment on lines +505 to +521

@thepastaclaw thepastaclaw Sep 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: Bound background warming on the shared BLS pool

Each quorum now contributes one job that computes every valid member's public-key share on the same 1–4-thread FIFO CBLSWorker pool used for signature verification, quorum-vector construction, aggregation, and DKG contribution verification. A fresh ScanQuorums() can enqueue up to keepOldConnections warmers—64 for LLMQ_60_75—so work submitted afterward remains behind the bulk warming queue, including callers waiting synchronously on BLS futures. The former dedicated thread isolated this background work. Preserve the thread reduction while limiting warming to one outstanding quorum at a time or otherwise prioritizing operational BLS jobs.

source: ['claude', 'codex']

@knst knst Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation logic is unchanged compare to develop, only its own thread is gone.
A warming job exists only for quorums this node holds a verification vector for: a masternode has those for the few quorums it is a member of, and only -watchquorums (a debug option) gets one per quorum.
Each job occupies one pool thread for one quorum, so anything pushed after it waits for a thread to free up, not for the whole batch. The pool has no priorities, and emulating the old thread with a single-flight queue would bring back the queue, the mutex and the interrupt this commit removes.

@thepastaclaw thepastaclaw Sep 9, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still applies (re-reviewed at e4f459d9): Your vector-availability argument narrows the affected workload, and I have corrected the earlier claim about production signature verification using AsyncVerifySig. The non-blocking scheduling concern remains for synchronous quorum/DKG operations: FIFO workers consume earlier queued warmers before later required work, including when a masternode has fetched additional vectors through -llmq-qvvec-sync.

}

// TODO: remove in v23
Expand Down
32 changes: 14 additions & 18 deletions src/llmq/quorumsman.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,12 @@
#include <unordered_lru_cache.h>

#include <sync.h>
#include <util/threadinterrupt.h>

#include <gsl/pointers.h>

#include <deque>
#include <atomic>
#include <map>
#include <memory>
#include <thread>

class CBLSSignature;
class CBLSWorker;
Expand Down Expand Up @@ -96,10 +94,7 @@ class CQuorumManager final
mutable Uint256LruHashMap<const CBlockIndex*, /*max_size=*/128> quorumBaseBlockIndexCache
GUARDED_BY(cs_quorumBaseBlockIndexCache);

mutable Mutex m_cache_cs;
mutable std::deque<CQuorumCPtr> m_cache_queue GUARDED_BY(m_cache_cs);
mutable CThreadInterrupt m_cache_interrupt;
mutable std::thread m_cache_thread;
std::atomic<bool> m_warming_interrupted{false};

public:
CQuorumManager() = delete;
Expand Down Expand Up @@ -135,20 +130,20 @@ class CQuorumManager final

// all these methods will lock cs_main for a short period of time
CQuorumCPtr GetQuorum(Consensus::LLMQType llmqType, const uint256& quorumHash) const
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps);
CQuorumCPtr GetQuorum(Consensus::LLMQType llmqType, const uint256& quorumHash, const CChain& chain) const
EXCLUSIVE_LOCKS_REQUIRED(::cs_main, !cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(::cs_main, !cs_db, !m_cs_maps);
std::vector<CQuorumCPtr> ScanQuorums(Consensus::LLMQType llmqType, size_t nCountRequested) const
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps);

// this one is cs_main-free
std::vector<CQuorumCPtr> ScanQuorums(Consensus::LLMQType llmqType, gsl::not_null<const CBlockIndex*> pindexStart,
size_t nCountRequested) const
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps);
std::vector<CQuorumCPtr> ScanQuorums(Consensus::LLMQType llmqType,
gsl::not_null<const CBlockIndex*> pindexStart,
size_t nCountRequested, const CChain& chain) const
EXCLUSIVE_LOCKS_REQUIRED(::cs_main, !cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(::cs_main, !cs_db, !m_cs_maps);

bool IsMasternode() const;
bool IsWatching() const;
Expand All @@ -175,29 +170,30 @@ class CQuorumManager final
CQuorumPtr GetCachedMutableQuorum(Consensus::LLMQType llmqType, const uint256& quorumHash) const
EXCLUSIVE_LOCKS_REQUIRED(!m_cs_maps);
void WriteContributions(const CQuorumPtr& quorum) const EXCLUSIVE_LOCKS_REQUIRED(!cs_db);
void QueueQuorumForWarming(CQuorumCPtr pQuorum) const EXCLUSIVE_LOCKS_REQUIRED(!m_cache_cs);
void QueueQuorumForWarming(CQuorumCPtr pQuorum) const;
/** Make queued and running warming jobs return; call before stopping the BLS worker pool */
void InterruptWarming() { m_warming_interrupted = true; }

private:
// all private methods here are cs_main-free
std::vector<CQuorumCPtr> ScanQuorums(Consensus::LLMQType llmqType,
gsl::not_null<const CBlockIndex*> pindexStart,
size_t nCountRequested, const CChain* chain) const
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps);
bool BuildQuorumContributions(const CFinalCommitmentPtr& fqc, const std::shared_ptr<CQuorum>& quorum) const;

CQuorumPtr BuildQuorumFromCommitment(Consensus::LLMQType llmqType,
gsl::not_null<const CBlockIndex*> pQuorumBaseBlockIndex,
bool populate_cache) const
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps);

CQuorumCPtr GetQuorum(Consensus::LLMQType llmqType, gsl::not_null<const CBlockIndex*> pindex,
bool populate_cache = true) const
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(!cs_db, !m_cs_maps);
CQuorumCPtr GetQuorum(Consensus::LLMQType llmqType, gsl::not_null<const CBlockIndex*> pindex,
const CChain& chain, bool populate_cache = true) const
EXCLUSIVE_LOCKS_REQUIRED(::cs_main, !cs_db, !m_cs_maps, !m_cache_cs);
EXCLUSIVE_LOCKS_REQUIRED(::cs_main, !cs_db, !m_cs_maps);

void CacheWarmingThreadMain() const EXCLUSIVE_LOCKS_REQUIRED(!m_cache_cs);
void MigrateOldQuorumDB(CEvoDB& evoDb) const EXCLUSIVE_LOCKS_REQUIRED(!cs_db);
};

Expand Down
9 changes: 5 additions & 4 deletions src/net.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4093,10 +4093,11 @@ bool CConnman::Start(CDeterministicMNManager& dmnman, CMasternodeMetaMan& mn_met
[this, connect = connOptions.m_specified_outgoing, &dmnman] { ThreadOpenConnections(connect, dmnman); });
}

// Initiate masternode connections
threadOpenMasternodeConnections = std::thread(&util::TraceThread, "mncon", [this, &dmnman, &mn_metaman, &mn_sync] {
ThreadOpenMasternodeConnections(dmnman, mn_metaman, mn_sync);
});
if (m_masternode_connections) {
threadOpenMasternodeConnections = std::thread(&util::TraceThread, "mncon", [this, &dmnman, &mn_metaman, &mn_sync] {
ThreadOpenMasternodeConnections(dmnman, mn_metaman, mn_sync);
});
}

// Process messages
threadMessageHandler = std::thread(&util::TraceThread, "msghand", [this] { ThreadMessageHandler(); });
Expand Down
5 changes: 5 additions & 0 deletions src/net.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1245,6 +1245,9 @@ friend class CNode;
SocketEventsMode socketEventsMode = SocketEventsMode::Select;
bool m_i2p_accept_incoming;
bool m_active_masternode = false;
//! Run the thread that opens connections to masternodes; only masternodes, quorum watchers and
//! CoinJoin mixing ever request them.
bool m_masternode_connections = true;
};

void Init(const Options& connOptions) EXCLUSIVE_LOCKS_REQUIRED(!m_added_nodes_mutex, !m_total_bytes_sent_mutex)
Expand Down Expand Up @@ -1283,6 +1286,7 @@ friend class CNode;
socketEventsMode = connOptions.socketEventsMode;
m_onion_binds = connOptions.onion_binds;
m_active_masternode = connOptions.m_active_masternode;
m_masternode_connections = connOptions.m_masternode_connections;
}

CConnman(uint64_t seed0, uint64_t seed1, AddrMan& addrman, const NetGroupManager& netgroupman,
Expand Down Expand Up @@ -1928,6 +1932,7 @@ friend class CNode;

/** Flag for activating masternode mode */
bool m_active_masternode{false};
bool m_masternode_connections{true};

SocketEventsMode socketEventsMode;
std::unique_ptr<EdgeTriggeredEvents> m_edge_trig_events{nullptr};
Expand Down
9 changes: 9 additions & 0 deletions test/functional/test_framework/test_node.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,13 @@ def __init__(self, i, datadir, extra_args_from_options, *, chain, rpchost, timew
"-debugexclude=leveldb",
"-debugexclude=rand",
"-uacomment=testnode%d" % i, # required for subversion uniqueness across peers
# Two threads keep the check queues and the RPC server concurrent; more only adds
# threads, and every thread costs ~0.9 MB of thread-local storage (BLS context).
"-par=2",
"-rpcthreads=2",
# The signature and script execution caches are allocated and zero-filled at startup
# whether or not anything is ever cached; 1 MiB still leaves 16384 entries each.
"-maxsigcachesize=1",
]
Comment on lines +119 to 124

@thepastaclaw thepastaclaw Sep 7, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocking: Gate -parbls for pre-v23 test binaries

TestNode applies this argument to explicitly versioned previous-release binaries as well as the current binary. Tests such as wallet_backwards_compatibility.py launch v21.1.1 and older releases, but -parbls was not registered until v23; those older ArgsManager implementations reject unknown command-line options as invalid parameters, so the nodes fail before startup. Keep the defaults supported by older releases unconditional and append -parbls=2 only for v23 or newer binaries.

Suggested change
"-par=2",
"-parbls=2",
"-rpcthreads=2",
# The signature and script execution caches are allocated and zero-filled at startup
# whether or not anything is ever cached; 1 MiB still leaves 16384 entries each.
"-maxsigcachesize=1",
]
"-par=2",
"-rpcthreads=2",
# The signature and script execution caches are allocated and zero-filled at startup
# whether or not anything is ever cached; 1 MiB still leaves 16384 entries each.
"-maxsigcachesize=1",
]
if self.version_is_at_least(23000000):
self.args.append("-parbls=2")

source: ['claude']

if self.mocktime != 0:
self.args.append(f"-mocktime={mocktime}")
Expand All @@ -138,6 +145,8 @@ def __init__(self, i, datadir, extra_args_from_options, *, chain, rpchost, timew
self.args.append("-logsourcelocations")
if self.version_is_at_least(22010000):
self.args.append("-loglevel=trace")
if self.version_is_at_least(23000000):
self.args.append("-parbls=2")

# Default behavior from global -v2transport flag is added to args to persist it over restarts.
# May be overwritten in individual tests, using extra_args.
Expand Down
Loading