dp2ptcs — System Documentation

This document explains the entire system end to end: what it is, the technologies it's built on, how every subsystem works, and how they all connect. It's written for a developer who needs to understand the whole system before changing anything in it.

---

1. What this system is

dp2ptcs is a peer-to-peer networking library written in Go. Each running instance is a node. Nodes:

• have a long-term cryptographic identity (an Ed25519 keypair, encrypted at rest),
• find each other over a Kademlia DHT (no central server, no directory),
• connect to each other directly over QUIC, punching through NAT with STUN and falling back to TURN relays when direct connectivity isn't possible,
• encrypt every application message end-to-end with a Double Ratchet (the same cryptographic construction used by Signal), layered on top of QUIC's own transport-level TLS 1.3 encryption.

In short: it's a Signal-style E2E-encrypted messaging layer, running over a self-organizing Kademlia network, transported over QUIC, with STUN/TURN NAT traversal — all as a single Go module with no external services required except an optional TURN server.

The pkg/api.Node interface is the entire public surface: Start, Stop, Send, Receive, ID, NATStatus. Everything else is internal implementation.

---

2. Technology stack

Concern	Technology	Where
Language / runtime	Go	whole repo
Transport protocol	QUIC (github.com/quic-go/quic-go)	internal/transport
Transport security	TLS 1.3, self-signed certs bound to Ed25519 identity	internal/transport/tls.go
Application-layer identity	Ed25519 (crypto/ed25519)	internal/crypto/identity.go
Application-layer E2E key exchange	X3DH (Extended Triple Diffie-Hellman, X25519)	internal/crypto/x3dh.go
Application-layer E2E encryption	Double Ratchet (Signal Protocol), HKDF-SHA256, ChaCha20-Poly1305	internal/crypto/double_ratched.go, kdf_chain.go, root_chain.go
Identity-at-rest encryption	Argon2id (KDF) + AES-256-GCM	internal/crypto/keystore.go
Identity file permission enforcement	POSIX file mode 0600 (Unix) / Win32 DACL via raw syscalls (Windows)	internal/crypto/file_store_unix.go, file_store_windows.go
Wire framing	Protocol Buffers (google.golang.org/protobuf) + varint length prefix	internal/messaging
Peer discovery / routing	Kademlia DHT (XOR distance, k-buckets, iterative lookup)	internal/dht
Multiplexing STUN + QUIC over one UDP socket	Custom demuxer, first-byte packet classification	internal/network/mux
NAT traversal (basic)	STUN (github.com/pion/stun), TURN (github.com/pion/turn/v2)	internal/network/nat
NAT traversal (advanced, not yet wired in)	ICE-style candidate pairing with signed candidate records	internal/transport/nat
Rate limiting	Token bucket, per source IP	internal/usecase/rate_limiter.go

---

3. Layered architecture

The codebase follows a rough Clean Architecture layering. Lower layers know nothing about higher layers:

pkg/api          — public interfaces (Node, Message, NATStatus). No implementation.
pkg/node         — the concrete Node implementation ("tacticalNode"). Wires everything together.
internal/usecase — application services: ConnectionManager, DiscoveryManager, NodeServer,
                    SessionManager, IdentityManager, rate limiting.
internal/domain  — pure entities and interfaces: Peer, Discoverer, CandidateRecord, validation.
                    No dependencies on crypto/network/transport.
internal/dht     — Kademlia: routing table, XOR distance, iterative lookup, the DHT wire RPC.
internal/handshake — the connection-time X3DH handshake protocol (turns a raw stream into
                    a SecureSession).
internal/messaging — wire format: Message struct, protobuf framing, handshake byte framing.
internal/transport — QUIC transport adapter + mTLS config + the ICE-style NAT subsystem.
internal/network  — the UDP demuxer (STUN vs QUIC) + STUN/TURN clients + legacy discoverer.
internal/crypto  — all cryptographic primitives: identity, X3DH, Double Ratchet, keystore,
                    per-OS secure file storage.
cmd/node         — the CLI entry point (`main.go`) that exercises the whole stack.

Dependency direction is strictly downward: pkg/node depends on usecase, which depends on domain/dht/crypto/transport/messaging, which depend on nothing above themselves. internal/crypto and internal/domain have no dependencies on any other internal package — they're the foundation.

---

4. Node lifecycle

4.1 node.New(opts ...Option) (api.Node, error)

Construction is synchronous and does no networking. In order:

1. Applies Option functions to DefaultConfig() (port 9000, 5s STUN/Bootstrap timeouts, 10s TURN timeout, 24h prekey rotation).
2. Falls back to the DP2PTCS_PASSPHRASE environment variable if no passphrase was set via options.
3. Validates config (Config.Validate()): port range, passphrase strength (via crypto.ValidatePassphrase), non-empty key path, positive timeouts.
4. Loads or creates the node's identity (usecase.IdentityManager.LoadOrCreate) — see §5.
5. Creates the in-memory session manager (§7.5) and the X3DH handshaker (§6).
6. Generates a self-signed mTLS tls.Config bound to the node's Ed25519 key (§8.2) and constructs the QUICTransport.
7. Builds the Kademlia RoutingTable (k=20) and DHTService, and a ConnectionManager (§9.1) wired to it. The routing table's stale-peer liveness check (SetPingFunc) is wired to ConnectionManager.Ping.
8. Builds the initial locally-advertised address list from cfg.PublicAddr / cfg.ListenAddr (skipping wildcard binds like 0.0.0.0:port), and constructs dht.NetworkRPCClient with it.
9. Builds usecase.DiscoveryManager (the real, production discoverer — see §10.2) and wires it into the ConnectionManager.
10. Builds the ICE-style candidate signaling service and the (currently unused in the main flow) HolePunchCoordinator / ConnectivityCoordinator — see §11.3.
11. Builds the appRouter (§9.4) and NodeServer (§9.3).

Nothing is listening on a socket yet. No network I/O has happened.

4.2 Start(ctx context.Context) error

This is where everything comes alive, strictly in this order (order matters — see the STUN/QUIC ordering note in §12.2):

1. Guards against double-start / start-after-stop under lifecycleMu.
2. Binds the UDP socket (net.ListenConfig{Control: setExclusiveAddrUse}.ListenPacket) — setExclusiveAddrUse is a Windows-only SO_EXCLUSIVEADDRUSE hardening no-op'd on Unix (§13).
3. Wraps that socket in mux.NewDemuxer (§12) — from this point on, the demuxer's readLoop() goroutine owns all reads from the real socket.
4. Builds the initial NAT-traversal candidate record set (refreshCandidateRecords, §11.3).
5. Starts the background prekey-rotation goroutine (§6.4).
6. If STUN servers are configured: runs one synchronous STUN pass (§10.1) — this must happen before QUIC starts listening, because STUN needs the socket's reads via the demuxer's STUNConn() and this ordering avoids races where QUIC's listener isn't draining its own queue yet. A 15-minute background ticker then keeps refreshing it.
7. If TURN servers are configured: allocates a relay (§10.3) and stands up a second, fully independent QUIC listener on the relay connection.
8. Starts QUIC listening on the demuxer's QUICConn() (the main listener) via n.transport.ListenOnConn(...), then starts NodeServer.StartWithListener in a background goroutine (§9.3).
9. For each configured bootstrap node, calls DiscoveryManager.Bootstrap (§10.2) — failures are logged, not fatal (a node can run in isolated mode).
10. Spawns a watcher goroutine that calls internal stop() when the context is cancelled.

4.3 Stop() error

Idempotent via sync.Once. Cancels the run context, closes the demuxer (which closes the real socket and both virtual conns), closes any TURN listeners, waits for all background workers, then closes the incoming channel.

---

5. Identity & key management

• Identity (internal/crypto/identity.go): an Ed25519 keypair. NodeID = SHA256(PublicKey) — a 32-byte value used everywhere as the canonical peer identifier (DHT keys, certificate pinning, session map keys). It is derived, not a separate secret.
• At-rest encryption (keystore.go): the private key is encrypted with AES-256-GCM, where the AES key is derived from the user's passphrase via Argon2id (128 MB memory cost, time cost 3, 4 threads — deliberately expensive to slow brute-force). ValidatePassphrase requires ≥12 characters and at least 3 of {upper, lower, digit, special}.
• File storage (file_store.go + per-OS files): FileIdentityStore implements IdentityStore{Save, Load}. Saving writes to a temp file then does an atomic rename. Loading first calls verifyFileSecure(path) before reading the file — this is a defense-in-depth check independent of the AES-GCM authentication:
  • Unix (file_store_unix.go): requires file mode exactly 0600.
  • Windows (file_store_windows.go): uses raw Win32 syscalls (SetEntriesInAclW, SetNamedSecurityInfo) to write a DACL granting access to the current user only, with inheritance explicitly disabled (PROTECTED_DACL). verifyCurrentUserOnlyACL (used by both production Load() and the test suite) checks the DACL is protected, contains exactly one ACE, and that ACE's SID matches the current user — this was fixed during development after CI caught that the original verifyFileSecure only checked file existence, not permissions, meaning a loosened ACL would silently be accepted at load time. See §14.6.
• IdentityManager.LoadOrCreate (usecase/identity_manager.go): tries Load; on ErrNoIdentityFound generates a fresh identity and persists it; any other error is propagated.

---

6. The X3DH handshake (connection-time E2E key agreement)

internal/handshake/handshake.go. This runs once per QUIC connection (not once per node — every time two peers establish a new connection, they redo this) and produces a crypto.SecureSession (the Double Ratchet, §7).

6.1 Key material per node

Each Handshaker holds, in addition to the node's long-term Ed25519 identity:

• a long-term X25519 identity key (localIdentityXPriv/Pub) — a separate DH-capable key, distinct from the Ed25519 signing key,
• a signed prekey (X25519), rotated periodically (§6.4).

6.2 The exchange

Both Initiate (dialer) and Respond (accepter) send a messaging.HandshakeExchange (§8.1): their Ed25519 identity pub, X25519 identity pub, current prekey pub, a fresh ephemeral X25519 keypair generated per-connection, an Ed25519 signature over all four, and a 5-minute expiry. Each side verifies the other's signature and expiry before proceeding.

6.3 X3DH math

Using the standard X3DH triple-DH construction (crypto/x3dh.go):

DH1 = IK_initiator × SPK_responder
DH2 = EK_initiator × IK_responder
DH3 = EK_initiator × SPK_responder
rootKey = HKDF-SHA256(0xFF*32 || DH1 || DH2 || DH3, info="Tactical-X3DH-Root-v1")

InitiateX3DH/RespondX3DH compute the same three DH products with operands swapped appropriately so both sides derive an identical rootKey, without ever transmitting it.

That rootKey seeds a brand-new crypto.DoubleRatchetSession (§7), keyed by the two ephemeral public keys exchanged in this handshake.

6.4 Prekey rotation

Handshaker.StartPrekeyRotation(ctx, interval) runs a ticker (default: cfg.Timeouts.PrekeyRotationInterval, 24h) that regenerates the X25519 prekey under a mutex. currentPrekey() takes a defensive copy so an in-flight handshake can't observe a torn/partially-rotated bundle.

---

7. End-to-end encryption: the Double Ratchet

internal/crypto/double_ratched.go, kdf_chain.go, root_chain.go, session.go.

7.1 The interfaces

type SecureSession interface {
    Encrypt(plainText []byte) (cipherText, dhPublicKey []byte, messageNumber uint64, previousChainLength uint32, err error)
    Decrypt(cipherText, remoteDHPubKey []byte, messageNumber uint64, previousChainLength uint32) (plainText []byte, err error)
}

DoubleRatchetSession is the production implementation. (SymmetricSession in secure_session.go also implements this interface but is a simpler two-fixed-chain variant used only in its own unit test — the live system always uses DoubleRatchetSession.)

7.2 Two-level ratchet

• Root chain (RootChain.Step): mixed forward with fresh DH shared secrets whenever the remote party's DH public key changes (i.e., on every new "epoch"). HKDF-SHA256 with the current root key as HKDF salt and the DH secret as the HKDF secret, producing a new root key + a new symmetric chain key. This is what gives post-compromise security — even if a chain key is ever exposed, the next DH ratchet step heals the session.
• Symmetric chains (KDFChain.Ratchet, one for sending, one for receiving): each call is a one-way HKDF step that destroys the old chain key and produces a fresh 32-byte message key. This is what gives forward secrecy — past message keys can't be recomputed from a later chain state.
• Initial chain assignment on session creation compares the two ephemeral public keys lexicographically (bytes.Compare) so both peers deterministically agree on which of the two chains is "send" vs "receive" without needing an explicit role flag.

7.3 Message encryption

Each message key seals the plaintext with ChaCha20-Poly1305 (12-byte random nonce prepended to ciphertext+tag). Encrypt ratchets the send chain, increments sendMessageNumber, and returns (ciphertext, currentDHSendPub, messageNumber, previousChainLength=0).

7.4 Decryption, epoch changes, replay & gap protection

Decrypt is the more intricate half:

1. Checks a skipped-key cache first (keyed by (remoteDHPubKeyHex, messageNumber)) in case this message arrived out of order and its key was already derived and stashed.
2. If the incoming remoteDHPubKey differs from the currently known one, this is a new epoch (the peer just DH-ratcheted): the message number must be exactly 1 (protocol violation otherwise), any messages skipped in the old epoch are ratcheted-and-cached up to previousChainLength, a fresh DH keypair is generated locally, both a new receive chain (from the peer's new key) and a new send chain (from the local new key) are derived via RootChain.Step, and both message counters reset.
3. Replay protection: an epochNumbers map (keyed by hex of the sender's DH pubkey) tracks the highest message number seen per epoch; anything ≤ that is rejected outright, independent of the skipped-key cache.
4. Gap bounding: MaxMessageGap = 1000 — a message number that would require ratcheting more than 1000 steps forward is rejected (ErrMessageGapTooLarge) to prevent a malicious/buggy peer from forcing unbounded CPU work.
5. Skipped-key eviction: capped at MaxSkippedKeys = 10000, evicted FIFO via a parallel queue slice, bounding memory even under sustained message loss.
6. Finally ratchets the receive chain forward to the target message number (caching each intermediate key as "skipped" along the way) and decrypts with ChaCha20-Poly1305; an authentication failure surfaces as "message authentication failed: potential tampering or out-of-sync ratchet".

7.5 Session storage: InMemorySessionManager

usecase/session_manager.go. Deliberately built as a single-goroutine actor: all operations (Get, Set, Use, Delete, DeleteIfCurrent) are sent as sessionRequest values over a channel to one dispatcher() goroutine, which is the only code that ever touches the map[string]*sessionMetadata. This sidesteps needing a mutex and makes "read-modify-write" style operations (like UseSession, used by Send — see §9.2) trivially race-free, since req.fn(meta.session) runs serialized inside the dispatcher itself.

• TTL eviction: sessions untouched for SessionTTL (1h) are dropped by a 15-minute cleanup tick.
• DeleteSessionIfCurrent(peerID, sessionID): guards against a race where connection A's teardown accidentally deletes a session that connection B (a newer, still-live connection to the same peer) has since replaced — each session gets a unique ID at SetSession time, and only a delete matching the current stored ID actually removes it.

---

8. Wire format

8.1 Handshake framing (internal/messaging/handshake.go)

Fixed-size, no length prefix needed — HandshakeExchange.WriteTo/ReadFrom write/read exactly 32+32+32+32+64+8 = 200 bytes: four 32-byte X25519/Ed25519 keys, a 64-byte Ed25519 signature, and an 8-byte little-endian Unix timestamp (expiry).

8.2 Application/control message framing (internal/messaging/message.go, protobuf_serializer.go)

type Message struct {
    SenderID            []byte // 32 bytes
    Type                MessageType // Packet | Command | Telemetry | DHT
    DHPublicKey         []byte // current ratchet epoch's DH public key
    Payload             []byte // Double Ratchet ciphertext
    MessageNumber       uint64
    PreviousChainLength uint32
}

ProtobufSerializer.Encode marshals this into pb.TacticalMessage (protobuf) and writes a varint length prefix followed by the encoded bytes onto the QUIC stream. Decode reads the varint, rejects anything over MaxMessageSize (1 MB hard cap — this is the guard that prevents a malicious length prefix from causing unbounded allocation), reads exactly that many bytes, unmarshals, and validates: SenderID must be 32 bytes, MessageNumber must be non-zero, PreviousChainLength capped at 10000, DHPublicKey must be 0 or 32 bytes.

Note: the ciphertext (Payload) travels inside this protobuf envelope — encryption happens at the Message level before Encode, decryption happens after Decode, both in the caller (NodeServer.handleStream, NetworkRPCClient.FindNode, tacticalNode.Send), never inside the serializer itself.

---

9. Peer connections & message flow

9.1 ConnectionManager (internal/usecase/connection_manager.go)

Bridges domain.Discoverer (logical peer → addresses) and transport.Transport (addresses → live connection):

• ResolvePeer: delegates straight to the discoverer.
• ConnectToPeer → DialAddresses: races all of a peer's known addresses concurrently (transport.Dial per address), returns the first successful connection, cancels the rest via a derived context.WithCancel. Every attempt and the eventual winner are logged at [DEBUG] level — this was added specifically to make NAT-traversal path selection observable (see §11).
• Ping: a 3-second-bounded ConnectToPeer + immediate close, used by the routing table's stale-peer liveness check (§10.3).

9.2 Sending a message: tacticalNode.Send (pkg/node/node.go)

1. Resolves to (hex PeerID) → *domain.Peer via DiscoveryManager.FindPeer (§10.2).
2. getPeerConnection: checks an in-process cache (peerConnections map[string]*peerConnection, keyed by hex NodeID) under peerMu. On a cache miss: dials via ConnectionManager.ConnectToPeer, opens one stream, runs the X3DH handshake as Initiator (§6) over that stream, stores the resulting session (SessionManager.SetSession), closes the handshake stream (it's single-purpose), caches the {conn, session} pair. A double-checked-locking pattern handles the race where two goroutines connect to the same peer simultaneously — the loser closes its redundant connection and reuses the winner's.
3. Opens a new stream on the (possibly cached) connection for the actual chat payload. If opening a stream on a cached connection fails (e.g., the peer dropped it), the cached entry is evicted and exactly one retry is attempted with a fresh connection.
4. Encrypts the payload via SessionManager.UseSession (so encryption is serialized through the session's owning dispatcher goroutine — see §7.5) producing (ciphertext, dhPubKey, msgNum, prevLen), wraps it in a messaging.Message{Type: TypePacket}, and writes it with ProtobufSerializer.Encode.

Note: a peer connection, once established, is reused across multiple Send calls — the handshake/session-establishment cost is paid once per peer, not once per message. This is separate from NetworkRPCClient.FindNode (§10.2), which dials and hand-shakes a fresh, ephemeral connection for every single DHT RPC.

9.3 Receiving: NodeServer (internal/usecase/node_server.go)

StartWithListener runs the accept loop (the production path — Start is the older direct-socket-binding variant, superseded once the demuxer was introduced but kept for standalone/testing use). Bounded by:

• connSem: a semaphore capping MaxConcurrentConnections (1000).
• PerIPRateLimiter (§9.5): rejects a new connection outright if the source IP is over its admission rate.

For each accepted connection (handleConnection, one goroutine per connection):

1. Accepts the first stream and runs the X3DH handshake as Responder (§6), which also returns the peer's Ed25519 identity public key from the handshake payload.
2. Cross-checks identity: computes expectedNodeID = SHA256(handshakeIdentityPub) and compares it to conn.PeerNodeID() (the NodeID derived from the QUIC connection's actual TLS certificate, §8.2 of the transport section below). A mismatch is logged as [SECURITY] and the connection is dropped — this binds the cryptographic handshake identity to the transport-layer TLS identity, so a peer can't present one identity at the QUIC/TLS layer and a different one inside the encrypted handshake.
3. Stores the session (SessionManager.SetSession), bounded by streamSem (MaxStreamsPerConnection, 100), then accepts and handles further streams on the same connection concurrently, each independently calling handleStream.
4. handleStream: reads one Message per loop iteration (bounded by io.LimitedReader at MaxMessageSize), decrypts via the session, calls the injected MessageHandler (this is appRouter.HandleMessage, §9.4), and if a response is returned, encrypts and writes it back on the same stream.
5. On connection teardown, the session is deleted via DeleteSessionIfCurrent (§7.5) so a stale session doesn't linger if the peer reconnects with a fresh handshake.

9.4 Application-level dispatch: appRouter (pkg/node/router.go)

HandleMessage switches on Message.Type:

• TypeDHT: unmarshals a pb.FindNodeRequest, calls DHTService.HandleFindNode (§10.4 — this is where inbound peers get added to the routing table), marshals the closest-peers response, returns it as a reply Message.
• TypePacket: pushes the decrypted payload onto tacticalNode.incoming (non-blocking; drops with a [WARN] log if the channel is full) as an api.Message, and returns an acknowledgement-style reply. (Note: this reply is currently constructed with Type: messaging.TypeDHT rather than TypePacket — almost certainly a copy-paste artifact rather than intentional, since chat replies are documented as "fire-and-forget" and the mismatched type has no observed effect on today's flow, but worth being aware of if anything ever starts branching on the reply's type.)
• Anything else is dropped with a [WARN] log.

9.5 Rate limiting (internal/usecase/rate_limiter.go)

Classic token bucket (RateLimiter): capacity + refill-per-second, tokens computed lazily on each Allow() call from elapsed wall-clock time (no background ticker needed for the bucket itself). PerIPRateLimiter keeps one bucket per source IP in a map, with its own 5-minute cleanup loop evicting buckets idle for 30+ minutes. NodeServer uses one instance (100 capacity, 10/sec) to gate new inbound connections per IP. The DHT RoutingTable has its own, separate, simpler per-IP limiter (§10.3) gating routing-table insertions per IP — these are two independent defenses at two different layers.

---

10. Kademlia DHT (peer discovery)

10.1 Core data structures (internal/dht)

• XORDistance(a, b) (distance.go): byte-wise XOR of two 32-byte NodeIDs — the Kademlia distance metric.
• RoutingTable (routing_table.go): 256 buckets (one per possible leading-zero-bit-count of the XOR distance, i.e., per "distance class"), each capped at k=20 peers.
  • AddPeer: rejects self-insertion, rejects if the source IP has exceeded 5 additions/minute (canAddFromIP), computes the bucket index, and:
    • if the peer ID is already present in the bucket, refreshes its lastSeen/status and moves it to the back (most-recently-seen) — but does not update its stored Addresses, even if the peer argument passed in this call carries a richer address list. This is a known limitation: an already-known peer's advertised address set is effectively frozen at first-seen. See §14.2.
    • if the bucket is full, checks whether the oldest entry has been silent for 15+ minutes; if so it's actively pinged (via the injected PingFunc, wired to ConnectionManager.Ping) and evicted if unreachable, making room for the new peer. Otherwise the new peer is rejected (classic Kademlia bucket behavior — long-lived peers are preferred over new ones).
  • ClosestPeers(targetID, count): flattens all buckets, sorts by XOR distance to targetID, returns the closest count.
• LookupTask (lookup.go): the iterative-lookup state machine. GetNextToQuery sorts the current shortlist by distance and returns up to α=3 unvisited peers per round (the standard Kademlia concurrency parameter); AddPeers merges newly-discovered peers into the shortlist without duplicates; GetClosest returns the top-k sorted results.

10.2 DiscoveryManager (internal/usecase/discovery.go) — this is the production discoverer

(There is also a internal/network/kademlia_discoverer.go implementing the same domain.Discoverer interface — it's a simpler, single-hop, non-iterative version, only exercised by its own unit tests. It is not constructed or used anywhere in pkg/node. Treat it as legacy/reference code, not part of the live system.)

• Bootstrap(ctx, bootstrapPeer): adds the bootstrap peer directly to the routing table, sends one FindNode RPC (§10.4) searching for the local node's own ID, and adds every peer in the response to the routing table. This is called once per configured bootstrap node at Start() time.
• FindPeer(ctx, targetID): first checks the local routing table for an exact ID match — if found, returns immediately without any network round-trip. This short-circuit matters: it means a peer's address information is exactly whatever was true the first time it was learned, with no automatic refresh (see §14.2 and §14.3 for the resulting limitation on bootstrap-target scenarios specifically). If no exact match, seeds a LookupTask from the current closest-known peers (plus the bootstrap peer as a fallback candidate) and runs the standard iterative Kademlia lookup: repeatedly take the next α=3 unvisited peers, FindNode them all concurrently (2-second-bounded each), feed responses back into the task, until no unvisited peers remain — every responsive peer along the way is added to the routing table.
• Announce/Refresh: both just call FindPeer(ctx, localID) — a Kademlia self-lookup, the standard way for a node to make its presence and current address known to its network neighborhood (since neighbors add the caller to their table as a side effect of answering — see §10.4).
• AddLocalAddress(addr): forwards to the RPC client (§10.5) so the given address is included in the SenderAddresses field of every future outbound FindNode RPC. Called by runSTUNRefresh and runTURNRefresh (§11) whenever a new externally-reachable address is discovered.

10.3 DHTService.HandleFindNode (internal/dht/service.go) — the inbound RPC handler

Runs on the receiving side of a FindNode RPC (wired through appRouter.handleDHT, §9.4). Two things happen:

1. Passive learning: merges the sender's self-reported senderAddresses with the actually-observed transport source address (mergePeerAddresses — dedups, and only includes the observed address if it round-trips through net.SplitHostPort), builds a domain.Peer, and calls RoutingTable.AddPeer. This is how the network learns about new nodes — there's no separate "announce" RPC type; simply answering any FindNode query is what teaches you about the asker.
2. Returns RoutingTable.ClosestPeers(targetID, k) — the requester's own responsibility is to recurse from there.

Important gap: HandleFindNode never includes the responder's own record (with its current self-advertised addresses) in the response — a node can't add itself to its own routing table (AddPeer explicitly rejects peer.ID == localID). Combined with §10.2's exact-match short-circuit, this means: a node currently has no protocol-level way to learn a peer's STUN/TURN-discovered addresses unless it either (a) was given that address directly, or (b) happens to be queried by that peer first (at which point the querying peer's own address is what gets learned, via point 1 above — not the responder's). See §14.3 for the full implication and possible fixes.

10.4 dht.NetworkRPCClient (internal/dht/network_client.go) — the outbound RPC transport

Implements dht.RPCClient.FindNode. For every single call, this:

1. Dials the target peer's known addresses via ConnectionManager.DialAddresses (the same Happy-Eyeballs-style race described in §9.1) — a brand new QUIC connection.
2. Opens a fresh stream and runs the X3DH handshake as Initiator — a brand new Double Ratchet session, used for exactly this one request/response pair, then discarded (the stream and connection are defer-closed at the end of the call).
3. Marshals a pb.FindNodeRequest{TargetId, SenderAddresses: c.localAddresses}, encrypts it, frames it as a messaging.Message{Type: TypeDHT}, writes it, reads back the response frame, decrypts, unmarshals into pb.FindNodeResponse, maps PeerInfo DTOs to domain.Peer entities (skipping and logging any malformed peer IDs rather than failing the whole lookup).

This is deliberately expensive per-call (full handshake every time) but keeps the DHT control plane using exactly the same authenticated, encrypted path as application traffic — there is no separate "cheap" unauthenticated discovery protocol.

AddLocalAddress(addr) just appends to c.localAddresses — no deduplication. A STUN refresh firing every 15 minutes with an unchanged mapped address will append the same string again on every tick; see §14.4.

---

11. NAT traversal

There are effectively two separate NAT-traversal subsystems in this codebase at two different levels of maturity. Understanding which one is actually live matters a lot.

11.1 STUN (live, in the main flow)

internal/network/nat/stun_traverser.go. STUNTraverser.GetExternalAddress(ctx, stunAddr, conn) sends a STUN binding request (github.com/pion/stun message construction) over the given net.PacketConn, loops reading responses until it gets one matching the request's transaction ID with a successful binding response class, extracts the XOR-MAPPED-ADDRESS attribute, and returns it as a string. Called from tacticalNode.runSTUNRefresh using n.demux.STUNConn() specifically — never the raw socket — so STUN traffic is cleanly demultiplexed away from concurrent QUIC traffic on the same physical socket (§12). Runs once synchronously at Start(), then every 15 minutes.

11.2 TURN (live, in the main flow, but architecturally separate from everything else)

internal/network/nat/turn_allocator.go. TURNAllocator.AllocateRelay wraps github.com/pion/turn/v2: creates a turn.Client bound to a dedicated, freshly-bound local UDP socket (not the node's main socket or the demuxer — TURN gets its own physical socket entirely), calls client.Listen() then client.Allocate(), and returns the relay's public address plus the turn.Allocation object, which itself implements net.PacketConn (reads/writes are transparently wrapped/unwrapped in TURN framing by pion).

tacticalNode.runTURNRefresh then does something structurally significant: it clones the QUIC transport (QUICTransport.Clone() — same TLS config, but independent pconn/quic.Transport state) and calls ListenOnConn on the relay connection, producing a second, fully independent QUIC listener running entirely over the TURN relay, in parallel with the main listener on the demuxed local socket. Both listeners run concurrently for the life of the node; tacticalNode.turnListeners/turnTransports track them for shutdown.

Pion's TURN authentication here is the long-term credentials mechanism (turn.ClientConfig{Username, Password}) — the TURN server must be configured with lt-cred-mech and matching static credentials, not the REST-API time-limited-secret flow.

11.3 The ICE-style candidate system (built, but not wired into the live connect path)

internal/domain/signaling.go, internal/transport/nat/{signaling_manager,ice,holepunch}.go. This is a considerably more sophisticated system:

• domain.CandidateRecord: a signed (ed25519.Sign over a canonical byte encoding), short-lived (default 60s TTL), typed (host / srflx / relay) address record.
• CandidateSignalingService.BuildCandidateSet: given host addresses, STUN-reflexive addresses, and TURN relay addresses, produces a signed CandidateRecord for each. tacticalNode.refreshCandidateRecords calls this after every STUN/TURN refresh, populating n.candidateRecords.
• CandidateSignalingService.FilterAndSortCandidates: verifies signatures + peer-ID binding + expiry on a remote peer's candidate set, and sorts by priority (host > srflx > relay — see typePriority).
• ConnectivityCoordinator.ProbeCandidatePairs (ice.go): the real ICE-style piece — builds the full local×remote candidate cross-product, sorted by combined priority, and races concurrent connection probes across all pairs, falling back to a relay address if nothing else connects within the deadline.
• HolePunchCoordinator.InitiateTraversal (holepunch.go): a simpler variant — verifies and sorts a remote candidate set, then dials all of them simultaneously and takes the first success.

Both HolePunchCoordinator and ConnectivityCoordinator are constructed in node.New and stored on tacticalNode, but neither is ever invoked from Start(), Send(), or getPeerConnection(). CandidatePathForPeer is a public-ish method on tacticalNode that would invoke ConnectivityCoordinator.ProbeCandidatePairs, but nothing in the automatic connect flow calls it. The actual live connect path (ConnectionManager.DialAddresses, §9.1) is a much simpler flat-address race with no type awareness and no priority ordering — see §14.1 for what this means in practice and what wiring it in would take.

---

12. The UDP demuxer — shared-socket multiplexing

internal/network/mux/demux.go. This is the piece that lets STUN and QUIC (and, later, x/sys/windows-cloned relay sockets are separate, but any future protocol sharing the main socket) coexist on one real UDP socket. It's also the subsystem with the most subtle history in this codebase — every fix here was hard-won, so the reasoning is preserved in detail.

12.1 Structure

type Demuxer struct {
    conn      net.PacketConn   // the real OS socket
    stunConn  *virtualConn     // net.PacketConn seen by STUN code
    quicConn  *virtualConn     // net.PacketConn seen by QUIC (quic.Transport)
    ...
}

One goroutine (readLoop) is the only code that ever calls ReadFrom on the real socket. For every packet, it classifies by first byte — STUN packets strictly start with a byte < 4 (per the STUN spec's message-type encoding), everything else is assumed to be QUIC — and delivers (virtualConn.deliver) it into the appropriate virtual connection's internal buffered channel (capacity 1024; full channels silently drop the newest packet rather than blocking readLoop, which would stall the other protocol too).

virtualConn implements net.PacketConn entirely over that channel, so from quic.Transport's point of view it's just talking to an ordinary packet connection — it has no idea a demuxer exists underneath.

12.2 Why STUN must run before QUIC starts listening

Start() deliberately runs the one-shot STUN pass (over demux.STUNConn()) before calling ListenOnConn(demux.QUICConn()). The demuxer's readLoop is already running at that point (it starts in NewDemuxer), so any QUIC packets arriving during that window just queue up harmlessly on quicConn's channel — nothing is lost, it's purely about not needing QUIC to be actively draining yet while STUN does its (also blocking, 5-second-bounded) request/response cycle.

12.3 Bugs found and fixed during development (all now fixed)

These are documented here because they're the kind of subtle, hard-to-repro issue that's likely to resurface in any similar shared-socket code added later:

1. Missing source address on read (the original, connection-breaking bug). ReadFrom returned the named return addr without ever assigning it from the queued packet — always nil. quic.Transport demultiplexes inbound datagrams and validates dial responses by peer address; with addr always nil, every QUIC handshake stalled to timeout. Fixed by returning pkt.addr (the address readLoop actually captured) instead of the unassigned zero value. Root-caused by temporarily bypassing the demuxer (pointing quic.Transport straight at the raw socket) and observing the handshake succeed instantly — proving the bug was specifically in the demuxer adapter, not QUIC or packet classification.

2. deliver()/Close() race. deliver() used to check v.closed and release the lock before sending on the channel, leaving a window where a concurrent Close() could close the channel out from under an in-flight send, panicking with send on closed channel. Fixed by holding v.mu for the entire check-and-send as one atomic critical section.

3. Shared, un-isolated write deadline. stunConn and quicConn wrap the same real socket, which only has one OS-level write deadline; the original code forwarded SetWriteDeadline straight to the real socket, so either virtual connection could clobber the other's deadline. Fixed by tracking writeDeadline per-virtualConn and serializing the actual realConn.SetWriteDeadline + WriteTo pair through a *sync.Mutex shared across every virtual conn on a given Demuxer (constructed once in NewDemuxer, passed to newVirtualConn). This design was chosen deliberately (over a simpler no-op) because the demuxer is meant to become a general shared-socket base layer for additional future protocols (e.g. media/RTP for a video-calling use case) — each new virtual conn gets correct, isolated deadline semantics for free by following the same newVirtualConn(conn, writeMu) pattern.

4. quic-go's "connection doesn't allow setting of receive buffer size" warning. quic.Transport tries to enlarge the OS receive buffer via a SetReadBuffer(int) error type-assertion; virtualConn didn't implement it, so quic-go silently ran without the larger buffer (harmless for light traffic, a real throughput/jitter risk under sustained load). Fixed by adding SetReadBuffer/SetWriteBuffer methods that forward directly to the real underlying socket (safe and correct: unlike the write-deadline case, OS buffer size is a genuine shared property of the fd, not something that needs per-virtual-conn isolation).

   Deliberately not implemented: SyscallConn(). quic.Transport also checks for this (via OOBCapablePacketConn) to unlock GSO/ECN batched reads/writes directly against the raw fd. If virtualConn exposed it, quic-go could bypass ReadFrom/WriteTo entirely and read/write the socket at the syscall level — which would skip the demuxer's STUN/QUIC classification in readLoop completely, silently breaking routing. This is guarded by a regression test (TestVirtualConn_SyscallConn_NotExposed) asserting the method is absent, specifically so a future throughput optimization doesn't reintroduce this without re-reading why it was left out.

5. TURN's relay listener still shows the receive-buffer warning, and that's expected, not a bug. The relay connection (turn.Allocation) is pion's own type, not virtualConn — it has no local kernel socket to resize (the relevant receive buffer, if any, lives on the coturn server, outside this process's control). This is an accurate signal from quic-go about a genuine limitation of relaying, not something fixable the same way.

12.4 Testing

demux_test.go covers PacketConn correctness (address propagation, Close() unblocking a pending ReadFrom, Close() not touching the shared real socket, deadline expiry both immediate and mid-block), packet-lifecycle behavior (multi-size packets, ordering, concurrent read/write across both virtual conns — tolerant of ordinary UDP loss but strict about payload/address correctness on whatever does arrive), and dedicated regression tests for the two concurrency bugs above. internal/transport/quic_demux_integration_test.go goes one level further and drives a real quic-go client/server handshake + stream exchange entirely through mux.Demuxer, on real UDP sockets — this is the test that would have caught the original address bug directly, since it exercises the exact production topology rather than testing either layer in isolation.

---

13. QUIC transport & mTLS identity binding

internal/transport/{quic_transport.go, tls.go, transport.go}.

• Transport interface: Dial(ctx, address, expectedNodeID) (Connection, error) and ListenOnConn(net.PacketConn) (Listener, error). Connection wraps a QUIC connection's stream multiplexing (OpenStream/AcceptStream) plus PeerNodeID(). MultiDialer (implemented by ConnectionManager) is the "race several addresses" abstraction used by both dht.NetworkRPCClient and nodeHolePunchDialer.
• GenerateNodeTLSConfig (tls.go): builds a self-signed X.509 certificate directly from the node's Ed25519 keypair (the cert is the identity — no CA). VerifyPeerCertificate enforces the presented cert's public key is Ed25519 and self-signature-valid; ClientAuth: tls.RequireAnyClientCert means QUIC's TLS handshake requires mutual authentication — every connection, in both directions, must present a valid certificate.
• QUICTransport.Dial: additionally sets a per-dial VerifyPeerCertificate closure that computes SHA256(peerCert.PublicKey) and compares it to the caller-supplied expectedNodeID — this is what pins a dial to a specific peer identity rather than merely "any validly self-signed cert."
• ListenOnConn: takes ownership of the given net.PacketConn (comment explicitly warns: any protocol-specific reads on that conn, like STUN, must be finished first — QUIC becomes the sole reader from that point on) and calls quic.Transport{Conn: conn}.Listen(...) with a server-side TLS config requiring any client cert (identity is verified per-connection at a higher layer, not pinned at listen time, since a listener doesn't know in advance who will dial in).
• quicConnection.PeerNodeID(): reads conn.ConnectionState().TLS.PeerCertificates[0], extracts the Ed25519 public key, hashes it — this is the transport-layer identity used for the cross-check in NodeServer.handleConnection (§9.3).
• QUICTransport.Clone(): same TLS config, fresh/nil pconn/tr — used exclusively to give the TURN relay listener (§11.2) its own independent transport state without disturbing the main listener's.

---

14. Known limitations & follow-up work

These are real, identified gaps — documented so they're deliberate backlog items, not silent surprises for whoever picks them up next.

14.1 The ICE-style candidate system isn't wired into the live connect path

HolePunchCoordinator/ConnectivityCoordinator (§11.3) are fully built, including host>srflx>relay priority ordering, but ConnectionManager.DialAddresses (the code path actually used by Send/Bootstrap/FindNode) is a flat, untyped address race. Wiring this in means deciding where in the connect flow candidate exchange happens (a new signaling RPC message type, most likely) and swapping DialAddresses for ConnectivityCoordinator.ProbeCandidatePairs at the right point.

14.2 RoutingTable.AddPeer doesn't refresh an existing peer's addresses

When a peer ID is already present in a bucket, only status/lastSeen are updated — the Addresses on the newly-passed-in domain.Peer are discarded. A peer's address set is effectively frozen at first-seen.

14.3 A node has no way to learn a peer's STUN/TURN addresses unless that peer queries it first

HandleFindNode never includes the responder's own record in its response (self can't be added to its own routing table), and DiscoveryManager.FindPeer short-circuits on an exact cached ID match without ever re-querying the network. Combined with §14.2, this means addresses discovered after a peer was first learned (e.g., STUN completing slightly later, TURN allocation) can't propagate to peers who only know the fixed bootstrap address and never dial in first. Candidate fixes: have HandleFindNode include a fresh self-record when appropriate, or have FindPeer periodically re-resolve even cached exact matches, or push an updated self-record over an already-open connection after the fact.

14.4 AddLocalAddress (both dht.NetworkRPCClient and, by extension, DiscoveryManager) has no deduplication

A 15-minute STUN refresh with an unchanged mapped address appends the same string again every tick, growing SenderAddresses unbounded on long-lived nodes. Needs a dedup (map-backed) or full-replace semantics instead of unconditional append.

14.5 Hot-path debug logging cost in the demuxer and connection dialer

demux.go's WriteTo/readLoop and ConnectionManager.DialAddresses log every packet/candidate synchronously via log.Printf (a global mutex + synchronous write). Fine for interactive chat traffic; a real latency/jitter risk once this carries sustained higher-rate traffic (e.g., media). Should be gated behind a debug flag before any such workload is added.

14.6 appRouter.handlePacket's reply message uses Type: messaging.TypeDHT

Likely a copy-paste artifact (chat replies are documented as fire-and-forget and nothing currently branches on this reply's type), but worth fixing for correctness/clarity before anything starts relying on reply message types.

14.7 (Resolved, documented for history) Windows identity-file permission verification

file_store_windows.go's verifyFileSecure originally only checked file existence, not the actual DACL — meaning a loosened permission set on Windows would silently be accepted by Load(). This was caught by a CI test failure (TestFileIdentityStore_LoadRejectsInsecurePermissions passing on Unix, failing on Windows) and fixed by moving the real ACL-verification logic (DACL protection, single-ACE, current-user-SID checks) out of test-only code and into production (verifyCurrentUserOnlyACL, shared by both Load() and the test helper). A second, related issue was fixed alongside it: the test's method of simulating "insecure permissions" (os.Chmod) is a no-op for ACLs on Windows (it only toggles the read-only attribute), so the test needed a Windows-specific ACL-loosening helper to actually exercise the check.

---

15. Configuration reference

pkg/node/config.go + options.go. All fields are set via Option functions passed to node.New.

Field	Option	Default	Notes
Port / ListenAddr	WithPort	9000 / 0.0.0.0:9000	WithPort keeps both in sync.
BootstrapID / BootstrapAddr	WithBootstrap	—	Single legacy-style bootstrap entry.
BootstrapNodes	WithBootstrapNodes	—	Preferred multi-bootstrap form.
PublicAddr	WithPublicAddr	—	Explicit externally-reachable address, if known in advance.
Passphrase	WithPassphrase	— / DP2PTCS_PASSPHRASE env fallback	Never defaulted; required by Validate().
KeyPath	WithKeyPath	node.key
STUNServers	WithSTUNServers	—	Enables §11.1.
TURNServers	WithTURNServers	—	Enables §11.2; each entry needs Address/Username/Credential.
Timeouts	WithTimeouts	STUN 5s, TURN 10s, Bootstrap 5s, PrekeyRotation 24h

---

16. Testing conventions

• Tests are co-located as *_test.go next to the code they cover, in the same package (white-box) except where a cross-package integration is being tested (e.g., internal/transport/quic_demux_integration_test.go imports both internal/network/mux and internal/transport).
• Concurrency-sensitive code (the demuxer, session manager, rate limiter) should always be run with -race: go test -race ./....
• The demuxer's test suite is a good model for testing a net.PacketConn adapter thoroughly: correctness of each interface method in isolation, packet-lifecycle edge cases (sizes, ordering, concurrent access, close-during-read, deadline-during-read), and a full real-protocol integration test on top, rather than only synthetic unit tests.

---

17. Entry point

cmd/node/main.go is a CLI wrapper: parses flags (-keyPath, -port, -bootstrapID, -bootstrapAddr, -stunServer, -turnServer, -turnUser, -turnPass, -chatWith), builds a node.New(...) with the corresponding Options, calls Start, and either runs a simple stdin chat loop (if -chatWith is set) or sits in passive/listen mode. examples/ contains smaller, more focused variants (basic usage, bootstrap peer/seed roles, chat sender/receiver) useful as minimal reference usage outside the CLI.