[POSIX] Preallocate exact capacity in marshalTlogEntryBundle - #1177
Conversation
5a12da9 to
d37da09
Compare
AlCutter
left a comment
There was a problem hiding this comment.
Ta, this is obviously much better than it was, but I wonder if changing the signature to take an io.Writer (or just a *byte.Buffer) and passing in currTile would be better still?
That would avoid the alloc in here entirely, although the "outside" bytes.Buffer will (continue to) have to realloc as it grows.
This method is only used by the mirror implementation. We could try to do some cleverness to avoid the allocation, but a single allocation probably isn't worth the overhead of doing this. Especially as this is 1:1 with a POSIX write so is negligible in the scheme of things. It's a shame this isn't used by Appender, as this would be a good performance win. I'll commit this as-is. |
Previously, marshalTlogEntryBundle allocated a fixed 16.78 MB slice on every call regardless of actual entry sizes. slices.Clip did not release the allocated backing memory, resulting in heavy heap churn and memory retention. For small leaves (e.g. 64B) this is about 400x faster. Benchmarks: // New BenchmarkMarshalTlogEntryBundle/entrySize=32B-24 183085 5965 ns/op 1373.33 MB/s 9472 B/op 1 allocs/op BenchmarkMarshalTlogEntryBundle/entrySize=128B-24 74240 17411 ns/op 1882.01 MB/s 40960 B/op 1 allocs/op BenchmarkMarshalTlogEntryBundle/entrySize=1024B-24 12888 94514 ns/op 2773.59 MB/s 270339 B/op 1 allocs/op // Old BenchmarkMarshalTlogEntryBundle/entrySize=32B-24 309 3693635 ns/op 2.22 MB/s 16785446 B/op 1 allocs/op BenchmarkMarshalTlogEntryBundle/entrySize=128B-24 352 3306127 ns/op 9.91 MB/s 16785409 B/op 1 allocs/op BenchmarkMarshalTlogEntryBundle/entrySize=1024B-24 349 3366554 ns/op 77.87 MB/s 16785412 B/op 1 allocs/op
d37da09 to
0623fa4
Compare
Previously, marshalTlogEntryBundle allocated a fixed 16.78 MB slice on every call regardless of actual entry sizes.
slices.Clip did not release the allocated backing memory, resulting in heavy heap churn and memory retention.
For small leaves (e.g. 64B) this is about 400x faster.
Benchmarks:
// New
BenchmarkMarshalTlogEntryBundle/entrySize=32B-24 183085 5965 ns/op 1373.33 MB/s 9472 B/op 1 allocs/op
BenchmarkMarshalTlogEntryBundle/entrySize=128B-24 74240 17411 ns/op 1882.01 MB/s 40960 B/op 1 allocs/op
BenchmarkMarshalTlogEntryBundle/entrySize=1024B-24 12888 94514 ns/op 2773.59 MB/s 270339 B/op 1 allocs/op
// Old
BenchmarkMarshalTlogEntryBundle/entrySize=32B-24 309 3693635 ns/op 2.22 MB/s 16785446 B/op 1 allocs/op
BenchmarkMarshalTlogEntryBundle/entrySize=128B-24 352 3306127 ns/op 9.91 MB/s 16785409 B/op 1 allocs/op
BenchmarkMarshalTlogEntryBundle/entrySize=1024B-24 349 3366554 ns/op 77.87 MB/s 16785412 B/op 1 allocs/op