Skip to content

[*] optimize flush() in pg sink - #1537

Merged
0xgouda merged 2 commits into
masterfrom
optimize-pg-sink-flush
Aug 28, 2026
Merged

[*] optimize flush() in pg sink#1537
0xgouda merged 2 commits into
masterfrom
optimize-pg-sink-flush

Conversation

@0xgouda

@0xgouda 0xgouda commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

We don't need to loop over all the rows in Measurements and read epoch_ns from each, because they typically all contain the same value extracted from now() in the same txn.

We don't have to loop over all the rows in a `Measurements` and
read `epoch_ns` from each because typically all of them will
contain the same value extracted from `now()` in the same txn.
@0xgouda
0xgouda requested a review from pashagolub August 28, 2026 12:08
@0xgouda 0xgouda self-assigned this Aug 28, 2026
@0xgouda 0xgouda added refactoring Something done as it should've been done from the start sinks Where and how to store monitored data labels Aug 28, 2026
@coveralls

coveralls commented Aug 28, 2026

Copy link
Copy Markdown

Coverage Report for CI Build 33176349266

Coverage decreased (-0.1%) to 87.878%

Details

  • Coverage decreased (-0.1%) from the base build.
  • Patch coverage: 5 of 5 lines across 2 files are fully covered (100%).
  • 8 coverage regressions across 1 file.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

8 previously-covered lines in 1 file lost coverage.

File Lines Losing Coverage Coverage
internal/sinks/prometheus.go 8 93.56%

Coverage Stats

Coverage Status
Relevant Lines: 6154
Covered Lines: 5408
Line Coverage: 87.88%
Coverage Strength: 1.0 hits per line

💛 - Coveralls

@pashagolub

Copy link
Copy Markdown
Collaborator

cool! do you have numbers on speed imrovements?

Comment thread internal/metrics/types.go Outdated
Co-authored-by: Pavlo Golub <pavlo.golub@gmail.com>
@0xgouda

0xgouda commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

cool! do you have numbers on speed imrovements?

Nope, I didn't benchmark it, but I would expect an improvement only for large setups with 100s of monitored sources, with each of them returning a lot of rows.

IMO, smaller setups won't be affected that much.

@pashagolub pashagolub left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Good job!

@0xgouda
0xgouda merged commit 25b0003 into master Aug 28, 2026
10 checks passed
@0xgouda
0xgouda deleted the optimize-pg-sink-flush branch August 28, 2026 13:59
@0xgouda

0xgouda commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

cool! do you have numbers on speed imrovements?

I did a small benchmark on my local machine mimicking 1000 sources, each with 80 active metrics with the same interval, and each metric returns 200 rows, and it was 7% faster:

goos: linux
goarch: amd64
pkg: github.com/cybertec-postgresql/pgwatch/v6/internal/sinks
cpu: Intel(R) Core(TM) i5-10500H CPU @ 2.50GHz
               │  old.txt   │              new.txt              │
               │   sec/op   │   sec/op    vs base               │
PgSinkflush-12   131.0 ± 7%   121.9 ± 4%  -6.92% (p=0.000 n=10)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactoring Something done as it should've been done from the start sinks Where and how to store monitored data

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants