Targets dial stale peer pod IPs on intra-data port 51083 after a rolling restart on Kubernetes; EC encoding fails silently

[garyz-inspirum]

Summary

After a rolling restart of the target StatefulSet on Kubernetes, targets continuously fail EC slice sends with i/o timeout while dialing peer pod IP addresses that no longer exist. The condition does not self-correct. A further rolling restart (one pod at a time, 60s apart) did not clear it; deleting all targets simultaneously did.

Environment

• aisnode v4.9, ais-init v4.9 (upgraded from v4.6)
• ais-operator, AIStore CR with proxySpec/targetSpec; portPublic: 51081, portIntraControl: 51082, portIntraData: 51083
• Kubernetes on-prem, Calico CNI (pod CIDR 10.42.0.0/16), 7 targets, 1 target per node
• Bucket ais://juicefs, EC 4+1, under active bulk ingest via the S3 API

Symptom

Target logs, repeating continuously:

E 14:13:46.200100 putjogger:526 failed to send ais://juicefs/juicefs-aistore/chunks/37/37280/37280470_0_8962 [ ec-resp[KyoloxVy=>VmvNRZMU] terminated [error when dialing 10.42.2.216:51083: dial tcp4 10.42.2.216:51083: i/o timeout ctx: dropping sobj-ais://juicefs/juicefs-aistore/chunks/37/37280/37280470_0_8962(size=2241)] ]
E 14:13:46.200204 putjogger:526 failed to send ais://juicefs/juicefs-aistore/chunks/37/37280/37280470_0_8962 [ ec-resp[KyoloxVy=>QeAqqucl] terminated [error when dialing 10.42.6.157:51083: dialing to the given TCP address timed out ctx: dropping sobj-ais://juicefs/juicefs-aistore/chunks/37/37280/37280470_0_8962(size=2241)] ]
E 14:13:46.200212 putjogger:566 Error while copying (data=4, parity=1) for "juicefs-aistore/chunks/37/37280/37280470_0_8962": ...
E 14:13:46.200268 putjogger:152 t[KyoloxVy]: failed to split ais://juicefs/juicefs-aistore/chunks/37/37280/37280470_0_8962, err: failed to send slice

Also observed on the same shared streams (different object named in the nested error than the one being sent):

E ... sendobj:142 ec-resp[KyoloxVy=>YbiMxLTR] sobj-ais://juicefs/.../chunks/37/37280/37280470_0_8962(size=2241): ec-resp[KyoloxVy=>YbiMxLTR] terminated [read /ais/data/@ais/juicefs/%ob/juicefs-aistore/chunks/37/37285/37285552_0_1183: file already closed ctx: dropping ...]

Evidence: the dialed addresses do not exist

The IPs being dialed belong to no pod in the cluster:

$ kubectl get pods -A -o wide | grep 10.42.2.216
$          # (no output)

Current target pod IPs at the time of those errors:

$ kubectl -n <ns> get pods -o wide | grep target
aistore-target-0   1/1  Running  0  10m     10.42.7.208   node01
aistore-target-1   1/1  Running  0  9m47s   10.42.3.220   node04
aistore-target-2   1/1  Running  0  8m37s   10.42.4.37    node02
aistore-target-3   1/1  Running  0  7m28s   10.42.6.158   node03
aistore-target-4   1/1  Running  0  6m19s   10.42.2.220   node05
aistore-target-5   1/1  Running  0  5m10s   10.42.1.184   node06
aistore-target-6   1/1  Running  0  4m1s    10.42.0.243   node07

10.42.2.216 was target-4's address in the previous pod generation; its current address is 10.42.2.220. Note the staggered pod ages — target-1 (9m47s) started ~3.5 minutes before target-4 (6m19s) was recreated.

Evidence: the Smap is correct and contains DNS names, not IPs

$ curl -s 'http://<proxy>:51080/v1/daemon?what=smap' | jq '.tmap | to_entries[] | {id: .key, pub: .value.public_net.direct_url, data: .value.intra_data_net.direct_url}'
{
  "id": "VmvNRZMU",
  "pub": "http://10.42.2.216:51081",
  "data": "http://aistore-target-4.aistore-target.<ns>.svc.cluster.local:51083"
}
...

Every intra_data_net.direct_url is a headless-service DNS name, and every public_net IP matched kubectl get pods -o wide at the time it was read. ais show cluster reported all 7 targets online throughout; intra-control (51082) traffic and keepalives were healthy for the entire period.

What did and did not resolve it

• Rolling restart, one target at a time with a 60s gap between pods: did NOT resolve. After completing the full sequence, targets were still dialing addresses from earlier pod generations. Each pod restarted early appeared to retain the addresses of peers that had not yet been recreated, and those peers subsequently changed IP.
• Deleting all 7 target pods simultaneously and letting them come back together: resolved it. All targets acquired their final IPs before any EC stream was established.

$ kubectl -n <ns> delete pod -l app.kubernetes.io/component=target --wait=false
$ kubectl -n <ns> wait --for=condition=Ready pod -l app.kubernetes.io/component=target --timeout=600s
$ for i in 0 1 2 3 4 5 6; do echo -n "target-$i: "; kubectl -n <ns> logs aistore-target-$i --since=3m | grep -c "i/o timeout"; done
target-0: 0
target-1: 0
target-2: 0
target-3: 0
target-4: 0
target-5: 0
target-6: 0

Impact

While the condition persists, EC encoding fails for affected objects (failed to split ... failed to send slice), but the client-side S3 PUT returns success — EC encoding is asynchronous. A bulk ingest therefore completes and reports success while producing objects whose 4+1 protection did not complete. No node goes NotReady, no pod restarts, and ais show cluster shows every target online, so there is no external signal that this is happening.

Questions

1. Is the intra-data (51083) peer endpoint expected to be re-resolved after a dial failure, or is it resolved once at stream-bundle creation and retained for the process lifetime?
2. Is there a supported way to force targets to re-establish intra-data streams against current peer addresses without restarting all targets simultaneously?
3. Is a rolling restart of the target StatefulSet (the default kubectl rollout / operator upgrade behaviour) a supported operation on Kubernetes, given pod IPs change on every restart?

Happy to provide additional logs, Smap dumps, or to re-run diagnostics on this cluster.