Migrate DB ledger read path to raw ledger bytes/view - #945
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
There was a problem hiding this comment.
Pull request overview
Migrates ledger reads toward zero-copy XDR views for issue #732, reducing unnecessary decoding of large ledger metadata.
Changes:
- Adds view-based ledger retrieval and streaming.
- Optimizes ledger-range and latest-ledger field extraction.
- Updates migrations, callers, mocks, tests, and SDK dependency.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
go.mod |
Bumps the Stellar SDK dependency. |
go.sum |
Updates SDK checksums. |
cmd/stellar-rpc/internal/db/ledger.go |
Implements view-based reads and prefix range queries. |
cmd/stellar-rpc/internal/db/ledger_test.go |
Tests view streaming and prefix fallback. |
cmd/stellar-rpc/internal/db/migration.go |
Lazily decodes views for migrations. |
cmd/stellar-rpc/internal/db/migration_test.go |
Adapts migration tests to views. |
cmd/stellar-rpc/internal/db/mocks.go |
Updates database reader mocks. |
cmd/stellar-rpc/internal/daemon/daemon.go |
Processes streamed ledger views during initialization. |
cmd/stellar-rpc/internal/methods/get_latest_ledger.go |
Builds responses directly from ledger views. |
cmd/stellar-rpc/internal/methods/get_latest_ledger_test.go |
Updates latest-ledger test reader. |
cmd/stellar-rpc/internal/methods/mocks.go |
Updates method-layer mocks for view APIs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // ledgerCloseTimePrefixBytes is the fast-path meta prefix fetched for range | ||
| // endpoints. Parsing falls back to the full blob if the header extends past it. | ||
| const ledgerCloseTimePrefixBytes = 1024 |
There was a problem hiding this comment.
This seems like a micro-optimization that isn't worth the add'l double-fetch complexity. The bulk of the time should be on encodes and decodes rather than the DB fetches themselves, which we mitigate via views already. Can you justify that this is worthwhile via some numbers?
There was a problem hiding this comment.
Yeah, good catch, I'll try to get some profiling on this later.
I'd intended to essentially eliminate the double-fetch path and just left that in for safety. I had figured that it was some wasted effort to fetch the entire DB when almost assuredly, only a small number of prefix bytes of the LCM were needed. But that can be discovered with numbers, too.
What
This PR covers some lingering ledger read path migrations to use ledger bytes/views where applicable. This has the immediate affect of optimizing
getHealthandgetFeeStats, though no changes to these handlers was required. Here are the functions modified:getLedgerRange/getLedgerRangeWithCache/getLedgerRangeWithoutCacheStreamLedgerRange(takes aStreamLedgerViewFnparameter instead of aStreamLedgerFnnow)daemon.mustInitializeStorage->db.MultiMigration.Applyhandle views instead of full LCMs. This is converted to an LCM indb.MultiMigration.ApplyAn adjacent optimization to the
getLedgerRange*functions was also made alongside these changes. In particular, these functions return only sequence numbers, but fetch entire LCM blobs (>1Mb). This is on the hot-path. To mitigate this, now, this behavior is a fallback -- we fetch the first 1Kb prefix of an LCM and use a view over it to fetch our sequences. This saves us some heavier fetching work and is relevant here because we no longer need to decode anything but the data that's relevant to the function. :-)Finally, the go SDK version
included was bumped, and accordingly, the abstraction of aRawLedger(a[]byteoverlay) was removed. This is because the SDK now includes aLedgerHeader()accessor, making theRawLedgerobsolete.Why
See #732. This is a component of adopting ledger views in RPC for query-path performance purposes.
Known limitations
N/A