NetBox registration fails with opaque "fetch failed" and no retry

[horner]

Symptom

Job #2115 (create-container.js --container-id=2122, 2026-09-06 02:38:12 UTC) ended with:

Registering container in NetBox (site: Phoenix DC)...
NetBox: VM creation failed (non-fatal): fetch failed

The container was created fine but has no NetBox VM record. The message gives no hint of what actually failed.

Investigation

• netbox_url = https://netbox.os.mieweb.org/, token valid. Authenticated curl from inside the manager container returns 200 in ~0.3s.
• The manager API server made successful node calls to the same endpoint at 02:33:11 and 02:44:29 UTC, bracketing the failure.
• The manager nginx access log (fronts netbox.os.mieweb.org → 10.17.26.141) has no /api/virtualization|dcim request at 02:38:0x–02:38:12. The request never reached nginx.
• No nginx/dnsmasq reload, and NetBox (CT 194) had not restarted.

So this was a network-level failure before HTTP (DNS / TCP / TLS), not a NetBox error. undici wraps those as a bare fetch failed and hides the real reason in err.cause, which we discarded.

Likely path: netbox.os.mieweb.org is a CNAME → os.mieweb.org → public 184.175.182.248, resolved by local dnsmasq → upstream 184.175.182.250/.251, then hairpin-NAT'd back in. A transient EAI_AGAIN or connect failure on that path is consistent with everything observed. createVirtualMachine also fires three lookups in parallel, so one blip aborts the whole registration.

Fix (code)

• Surface err.cause.code and the request URL in the thrown error.
• Retry network-level fetch failures (3 attempts, backoff); do not retry HTTP error responses.

Follow-up (infra, out of scope here)

Pointing netbox_url at the internal https://netbox.cluster.mieweb.org (10.17.26.141) would avoid external DNS and hairpin NAT, but NetBox currently serves a self-signed cert (CN=netbox.os.mieweb.org), so the manager would need to trust it first.