Add btcd-compatible JSON-RPC/WebSocket endpoint - #838
Open
eynhaender wants to merge 16 commits into
Open
Conversation
New endpoints/test_btcd_rpc.py covering the btcd JSON-RPC/websocket compatibility interface, modeled on test_bitcoind_rpc.py's request/response pattern and test_electrum_subscriptions.py's persistent-connection/ notification-reading pattern. No new pip dependency: the websocket client (WebSocketConnection) is a minimal stdlib-only RFC 6455 implementation (handshake, masked text frames, multi-frame message reassembly), matching this suite's existing reference for hand-rolled wire protocol over a library (see Electrum's raw-socket client). Tests are split into two kinds: - Real assertions (no marker) — authenticate, session, notifyblocks/stopnotifyblocks, chain methods over http post, deprecated methods and stop staying rejected, response envelope behavior. A failure here is a regression. - @pytest.mark.xfail(strict=False) — development targets for what isn't implemented yet: chain methods bridged into the ws connection itself (phase B), the wired not-yet-implemented stubs (notifynewtransactions, loadtxfilter, rescanblocks, ...), and the not-yet-wired phase B/C methods (getinfo, getdifficulty, createrawtransaction, ...). Expected to fail now; flips to XPASS once implemented, which is the cue to tighten the assertion and drop the marker. conftest.py: --btcd-host/--btcd-port/--btcd-username/--btcd-password options and btcd_config fixture. utils.py: DEFAULT_BTCD_PORT. README.md: btcd section (config sample, options, coverage, debug/trigger env vars) alongside the existing per-endpoint documentation.
New test/protocols/btcd/{btcd_rpc.cpp,btcd_setup_fixture.{hpp,cpp}}, in the style of test/protocols/bitcoind: a real websocket connection to a mock ten-block store, covering authenticate, session, notifyblocks/stopnotifyblocks, every not_implemented stub, response id echo across multiple requests on one connection, and unknown-method handling (json-rpc error reply, connection stays alive for the next request).
Adds channel_btcd (channel_http + per-connection auth state, preselects the json-rpc request parser for websocket frames) and protocol_btcd_rpc (inherits protocol_bitcoind_rpc for the standard chain handlers, adds a second dispatcher for btcd's session/notification/filter/admin extension methods over dispatch_websocket). Live: authenticate (validates against configured btcd.username/password, or no-op success if none configured), session (returns the channel identifier), notifyblocks/stopnotifyblocks (chase::organized/reorganized -> blockconnected/blockdisconnected, positional [hash, height, time] per real btcd wire format). Stubbed not_implemented: stop (no secure remote-shutdown path exists), notifynewtransactions/stopnotifynewtransactions (no mempool in v4), loadtxfilter/rescanblocks (phase B), the deprecated notifyreceived/ notifyspent/rescan family. Extends interfaces/btcd.hpp with a stop method. Wiring: channels.hpp, protocols.hpp, sessions.hpp, server_node.hpp/.cpp (start_btcd chained after bitcoind), parser.cpp ([btcd] config section mirroring [bitcoind]), builds/gnu/Makefile.am. Auth over websocket: basic auth is a per-http-request header, which ws data frames cannot carry (only the upgrade handshake has headers, and that request never reaches the auth check -- see channel_http's error::upgraded short-circuit). channel_btcd::unauthorized() bypasses the inherited per-request check for websocket traffic; the plain http post path keeps it unchanged. Enforcement over ws moves into dispatch_websocket instead: every method other than authenticate is rejected until the connection has authenticated. handle_authenticate's reject path also no longer calls stop() synchronously alongside the error send -- that raced the async write and could close the connection before the error response flushed. It's now threaded through as the SEND completion reason, so the close only happens once the write actually finishes. Known gap: standard chain methods (getblockcount etc.) remain reachable only via plain http post on the same endpoint, not yet bridged into the ws dispatcher -- see the phase B TODO on protocol_btcd_rpc.hpp.
conn fixture now authenticates up front when the server has credentials configured, matching what a real client must do before calling anything else over ws (every method but authenticate is now rejected until then). Without this, most of the suite degraded to xfail/failed once credentials were actually configured on the server, since none of the individual tests called authenticate themselves. http_rpc() now sends HTTP Basic Auth when credentials are configured, for the same reason on the plain-post path (which keeps the normal per-request check, unlike ws). test_response_id_matches_request no longer hardcodes absolute request ids 0/1: the fixture's own authenticate call consumes an id first when credentials are configured, so only the relative ordering (each id is exactly one more than the last) is a stable guarantee. Verified against a live bs_btcd instance with btcd.username/password configured: 19 passed, 2 skipped, 20 xfailed, 0 failures.
…ake stage).
session_handshake<protocol_btcd_auth, protocol_btcd_rpc> (tried in the two
preceding commits, now dropped) modeled btcd's auth on electrum/p2p's version
negotiation, but that pattern assumes the handshake can only complete by
reading a real message. btcd's doesn't: with no credential configured it
completes without reading anything, leaving a real async gap between
"handshake decided it's done" and protocol_btcd_rpc actually subscribing its
handlers. Harmless for websocket (the upgrade round-trip masks it), but a
plain HTTP client can and reliably did land a request in that gap. Real btcd
has no such gap because it has no handshake protocol at all.
Revert session_btcd to session_server<protocol_btcd_rpc> (matching bitcoind
and real btcd) and move 'authenticate' back into protocol_btcd_rpc as an
ordinary extension method, but with the real semantics this time:
- Digest-based credential check (sha256("Basic " + base64(user:pass)), same
scheme as config::credential) via options_.authorized(), instead of a
plaintext username/password comparison.
- btcd.username/btcd.password collapse into the same 'btcd.credential =
user:pass[:method,...]' shape bitcoind already uses, so a credential can be
scoped to a method allowlist -- channel_btcd::permitted() enforces that
scope for both http and ws (ws basic-auth-on-upgrade and in-band
authenticate are two alternative ways to satisfy it, matching real btcd's
own "authenticate is disallowed once basic auth is already established").
- send_btcd_error/send_btcd_rpc thread a close_reason through to
handle_complete, so a failed authenticate's error reaches the client before
the connection closes, without racing the async send.
docs/btcd-endpoint.md documents why the handshake variant was reverted.
Boost.Test (640 cases) and the Python endpoint suite pass against a live
server both with and without a configured credential.
Some endpoint tests (e.g. test_blockconnected_notification) wait on a real external event -- an actual new block -- up to --subscription-timeout, which made a normal pytest run slow and made CI/local runs flaky when no block happened to arrive in time. Mark those @pytest.mark.slow and skip them by default; --run-slow opts back in for a real (if longer) end-to-end check.
Reaches standard chain methods over the ws connection (not just plain http post), adds getcurrentnet, implements loadtxfilter with filtered block notifications and rescanblocks (base58 + segwit address/outpoint matching), and wires the remaining generic-tooling Phase C methods (getdifficulty, getinfo, getnettotals, getnetworkhashps, createrawtransaction, decoderawtransaction, decodescript, validateaddress, help). See docs/btcd-endpoint.md for details.
Covers ws-bridged chain methods, getcurrentnet, loadtxfilter/ filteredblockconnected/disconnected/rescanblocks (including a constructed p2kh test block for address/outpoint matching), and the Phase C generic RPC methods. Adds a notify() fixture helper to synthesize chase events without a live p2p sync.
Flips ws-bridged chain methods and getcurrentnet from xfail to real tests, adds loadtxfilter/rescanblocks/filteredblockconnected coverage (the latter marked slow, needs --run-slow) and Phase C method tests, and adds a MAINNET_MAGIC reference constant.
Marks the ws-bridging gap, getcurrentnet, loadtxfilter/filtered notifications/rescanblocks, and the Phase C filler methods as implemented, corrects the lnd-compatibility notes against btcd's real wire methods (filteredblockconnected/relevanttxaccepted, not the deprecated recvtx/redeemingtx pair), and drops the stale address_enabled guard note for rescanblocks.
A real lnd integration test found that btcwallet's chain.RPCClient calls getinfo over plain http post immediately on connect, before any ws traffic. getinfo (btcd-only) had no path there and unrecognized methods dropped the connection outright, so lnd retried forever. Adds protocol_bitcoind_rpc::dispatch_extension (post-side mirror of the existing ws-side dispatch_rpc fallback), overridden by protocol_btcd_rpc to try btcd_dispatcher_, and softens handle_receive_post to reply with a soft error instead of dropping the connection when a method is unrecognized by both dispatchers.
Adds btcd_setup_fixture::http_rpc (a plain-post connection alongside the existing ws one, mirroring bitcoind_setup_fixture's own helper) and a regression test calling getinfo over it; bumps the fixture's btcd.connections to 2 for the two simultaneous connections. Mirrors the same case into the Python acceptance suite.
lnd's chainreg.backendSupportsTaproot only checks for a "taproot" key under bip9_softforks in getblockchaininfo (falling back to the getdeploymentinfo RPC, which has no equivalent in real btcd, only if that's absent) -- confirmed against lnd's own source. Real btcd's own getblockchaininfo includes this too, so this is shared, correct behavior for both the bitcoind and btcd endpoints, verified end to end against a real lnd instance getting past its chain-backend setup.
A real lnd integration test found this blocking wallet chain-sync
bootstrap: btcwallet's own rpcclient connection (distinct from lnd's
own chain.RPCClient, which uses getbestblockhash) calls this btcd
extension method. Confirmed against btcd's own btcjson/rpcclient
source that it takes no params and returns {hash, height}, unlike
getbestblockhash's bare hash string.
Real btcd's own handleRescan skips scanning entirely and reports immediate completion when given no addresses/outpoints to watch -- exactly what btcwallet's own rpcclient sends to bootstrap its initial sync starting point (found via a real lnd integration test, previously a hard failure: "18: not_implemented"). Validates beginblock against the local chain and rejects unknown hashes, matching real btcd. A non-empty address/outpoint list remains not_implemented: it would need a real historical block scan (recvtx/redeemingtx notifications, chunked progress, reorg recovery) that no observed real caller needs, since loadtxfilter/rescanblocks already cover actual address watching. Verified end-to-end against a real lnd 0.21.1 binary: it now reaches "Chain backend is fully synced!" against this server.
The design doc and several code comments still described the endpoint in terms of its original three-phase rollout plan, which the implementation has long since outgrown (the real lnd integration work this session -- POST-side dispatch bridging, taproot signaling, getbestblock, rescan -- doesn't map cleanly onto any of the three phases). Reorganized docs/btcd-endpoint.md's Method Scope and lnd Compatibility sections around what's actually implemented vs. permanently stubbed instead, folded in the previously-undocumented recent additions, and recorded that lnd has been verified reaching full chain sync end-to-end against this endpoint. Dropped the matching Phase/B0/B1/B2 labels from code comments and the two endpoint test suites' stale coverage descriptions (endpoints/README.md's Coverage section had drifted furthest -- it still described loadtxfilter, rescanblocks, getcurrentnet, and the ws-bridged chain methods as not-yet-implemented xfail targets, all of which have been implemented and asserted-on for some time).
eynhaender
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
btcd-compatible endpoint (JSON-RPC 1.0 over WebSocket + plain HTTP POST, port 8334 by convention), following the pattern established by thebitcoindendpoint. Full design rationale, method scope, and an lnd/btcwallet compatibility analysis are indocs/btcd-endpoint.md.protocol_btcd_rpcinheritsprotocol_bitcoind_rpc(all standard chain handlers —getblockcount,getblock,getrawtransaction,sendrawtransaction, etc. — come along unchanged, reachable over both transports) and adds a second dispatcher for btcd's own session/notification/filter/admin extension methods.Verified end-to-end against a real lnd binary
Ran an actual lnd 0.21.1-beta binary against a libbitcoin-server instance running this endpoint (
--bitcoin.node=btcd) — it reachesChain backend is fully synced!: wallet opened/unlocked, chain-sync bootstrap completed, headers walked to the live chain tip. Every gap below was found this way (not just by reading btcd's source), each confirmed fixed by a subsequent clean run one step further than the last.Implemented
authenticate,session(real per-channel id —btcwalletuses it to detect reconnect-to-a-fresh-server).notifyblocks/stopnotifyblocks→blockconnected/blockdisconnected(verified wire format).loadtxfilter(base58 + bech32/bech32m) +filteredblockconnected/filteredblockdisconnected+rescanblocks— the mechanism bothbtcwalletand lnd's own notifier use for funding-confirmation and breach/force-close detection.getcurrentnet,getbestblock({hash, height}, a distinct btcd extension fromgetbestblockhash— needed bybtcwallet's own separate rpcclient connection during chain-sync bootstrap).rescan(deprecated upstream) for its empty-addresses/empty-outpoints case — the exact callbtcwalletmakes to bootstrap its initial sync starting point, matching real btcd's own "skip scanning" shortcut.getblockchaininfonow reportsbip9_softforks.taproot— required by lnd'schainreg.backendSupportsTaprootbefore it will treat any btcd/bitcoind backend as usable.getinfo) over POST immediately on connect, before the persistent ws connection subscriptions need.getdifficulty,getinfo,getnettotals,getnetworkhashps,createrawtransaction,decoderawtransaction,decodescript,validateaddress,help.Permanently stubbed (
not_implemented)stop— no secure remote-shutdown path exists.notifynewtransactions/stopnotifynewtransactions, andrescan's non-empty-address/outpoint case — blocked on a v5 mempool / no observed real caller needs a full historical scan (loadtxfilter/rescanblocksalready cover real address watching).notifyreceived/stopnotifyreceived/notifyspent/stopnotifyspent.submitblock, mining, and peer-management methods — out of scope.Test plan
C++ (
test/protocols/btcd/)Python (
endpoints/test_btcd_rpc.py)Real lnd integration