qkc/slave 04: implement baseConn and XshardConn compatibility layer - #32
qkc/slave 04: implement baseConn and XshardConn compatibility layer#32iteyelmp wants to merge 18 commits into
Conversation
# Conflicts: # qkc/cluster/wire/messages_test.go
qzhodl
left a comment
There was a problem hiding this comment.
Please update the PR description to state explicitly whether a full Codex 5.6 review was run against the complete diff before submission, for example: Full Codex 5.6 review completed: yes/no.
My first Codex review of this PR found multiple correctness and lifecycle issues that the existing tests did not expose, including successful acknowledgements for discarded xshard data, unchecked response error codes, stale multi-shard routes, and timeout/late-response teardown. Please run a complete Codex review before submitting code, address or explicitly document its findings, and disclose the review status and any intentional incomplete stubs in the PR description.
| } | ||
| return res.frame, nil | ||
| case <-ctx.Done(): | ||
| return nil, fmt.Errorf("rpc timeout: %w", ctx.Err()) |
There was a problem hiding this comment.
[P2] On context timeout, the deferred cleanup removes this RPC from pending; a later valid response then reaches the unknown-RPC path in readLoop and closes an otherwise healthy connection. Python retains the map entry until the response or connection close and ignores delivery to a cancelled future. In Go, please retain a bounded cancelled-RPC tombstone until a late response is consumed (or close the connection deliberately on timeout), and add a slow-response regression test.
| // closeMu → stateMu (Close) | ||
| // | ||
| // pendingMu and stateMu are never held together; readLoop only holds pendingMu. | ||
| type baseConn struct { |
There was a problem hiding this comment.
baseConn is explicitly intended to be shared by XshardConn and the future MasterConn, so putting the generic transport/RPC engine under slave gives it the wrong ownership boundary. Please keep wire limited to framing and schemas, move the generic connection layer to a cluster-level package such as qkc/cluster/conn or qkc/cluster/protocol, and keep XshardConn/XshardPool under slave.
| localFullShardIDList []uint32 | ||
|
|
||
| // peer identity state, protected by its own mutex (not baseConn.closeMu). | ||
| stateMu sync.Mutex |
There was a problem hiding this comment.
XshardConn.stateMu shadows the embedded baseConn.stateMu, although the two mutexes protect unrelated state. I do not see a current lock cycle here, but this makes future locking mistakes unnecessarily easy. Please rename this mutex to identityMu and document that it protects remoteID and remoteFullShardIDList.
| // SendRPCMeta sends a request with the given metadata and waits for the response. | ||
| // XshardConn uses zero metadata (0-byte wire format). | ||
| // MasterConn uses ClusterMetadata{Branch, ClusterPeerID} (12-byte wire format). | ||
| func (c *baseConn) SendRPCMeta(ctx context.Context, opcode byte, payload []byte, meta wire.ClusterMetadata) (*wire.Frame, error) { |
There was a problem hiding this comment.
rpc_id comes from an atomic counter, but the frame is written later under a different lock (transport.writeMu), so two senders can put N+1 on the wire before N. The peer's monotonic validator rejects it and tears down the connection. Python is immune — no await between self.rpc_id += 1 and write_command on a single-threaded loop. Reachable in normal operation: VerifyAndAddToShards indexes one connection under several shards. Fix: allocate the id inside the write critical section. Suggested test:
func TestSendRPC_ConcurrentSendsPreserveRPCIDOrder(t *testing.T) {
client, server, cleanup := newTestConnPair(t)
defer cleanup()
server.Start()
client.Start()
payload, err := serialize.SerializeToBytes(&wire.PingRequest{
ID: []byte("client"), FullShardIDList: []uint32{0x00010001},
})
if err != nil {
t.Fatal(err)
}
const n = 200
var wg sync.WaitGroup
start := make(chan struct{})
for i := 0; i < n; i++ {
wg.Add(1)
go func() {
defer wg.Done()
<-start
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
client.SendRPC(ctx, byte(wire.ClusterOpPing), payload)
}()
}
close(start)
wg.Wait()
select {
case <-server.WaitUntilClosed():
t.Fatalf("connection torn down: %d concurrent RPCs emitted rpc_ids out of order", n)
case <-time.After(500 * time.Millisecond):
}
}
|
|
||
| total := 0 | ||
| for _, conns := range p.conns { | ||
| total += len(conns) |
There was a problem hiding this comment.
OutboundSize is misleading in two ways: after WatchAndIndex, conns contains both outbound and verified inbound connections, and summing the per-shard slices counts route entries rather than unique connections, so one connection serving N shards is counted N times. Please either rename this metric to IndexedRouteCount/RouteCount, or deduplicate connection pointers and expose a true ConnectionCount. If direction-specific counts are required, the pool must retain direction metadata instead of discarding it after indexing. The Targets comment should likewise describe routable shard IDs rather than outbound targets.
This PR introduces the foundational connection layer for cluster communication.
The main goal is to reproduce the Python cluster connection behavior in Go,
while keeping business logic migration out of scope.
Implemented
baseConn
Introduces the common connection abstraction used by future cluster
communication components.
Implemented:
XshardConn
Implements slave-to-slave communication on top of baseConn.
Implemented:
XshardPool
Adds connection management for xshard peers.
Implemented: