Skip to content

Address retirement on port-release timeout is one-shot for the process lifetime; with a 1-address pool the run deadlocks in getNextAvailableLoopbackAddress even though retirement is meant to be a parking action #27

Description

@deliciousmonster

teardownHarper waits DEFAULT_PORT_RELEASE_TIMEOUT_MS (default 5s) for all fixed Harper
ports to free, and on expiry it does not release the loopback address
(dist/harperLifecycle.js:618-629 in 0.7.1): the slot stays parked under the current
process's PID and the warning says "The slot will be reclaimed when this process exits."
There is no later re-check. Retirement is permanent for the process lifetime, however briefly
the ports were actually held.

That interacts badly with two facts:

  • Under --isolation=none (harper-integration-test-run) every suite in the run shares one
    process, so "reclaimed when this process exits" means "never, for the rest of this run".
  • On stock macOS only 127.0.0.1 is bindable (127.0.0.2+ need aliases), so a practical local
    pool is one address (HARPER_INTEGRATION_TEST_LOOPBACK_POOL_START=1, COUNT=1).

After one retirement, the next suite's startHarper enters
getNextAvailableLoopbackAddress and spins forever: the free-slot scan finds nothing and
removeDeadProcessesFromPool (dist/loopbackAddressPool.js:414-425) cannot reclaim a slot
whose holder PID is the caller's own live process. The wait is unbounded and silent.

Measured sequence, one macOS host, 2026-08-14, three consecutive
npm run test:integration invocations of the same two-file suite:

Run (local time) PORT_RELEASE_TIMEOUT_MS Suite 1 Then
15:42 5000 (default) green, 15.2s total wait expired, warning logged, address retired
15:53 5000 (default) green, 13.4s total same
16:05 60000 (override) green, 68.1s total 60s wait also expired; suite 2 never acquired an address; the harness had to be killed externally and $TMPDIR/harper-integration-test-loopback-pool.json reset by hand

Raising the timeout 12x changed only the stall length, which separates the two defects: why
the ports were held is its own issue (on this host an unrelated resident MQTT broker held
1883, filed separately), but the one-shot retirement is what turns any expiry into a wedged
run. In the case the code comments actually contemplate (a slow Harper child that exits a few
seconds after the deadline), the ports free and the address stays retired anyway; that case
follows directly from the code path (no re-check exists) and is expected rather than
separately measured here.

A second, related defect: the pool persists holder PIDs in
$TMPDIR/harper-integration-test-loopback-pool.json across processes
(dist/loopbackAddressPool.js:23). A run killed with SIGKILL leaves its PID parked; after
the 16:05 run was killed, the pool file still carried that run's slot state and was reset to
[] by hand before anything else ran (measured). What happens without that reset is a code
read, not a measurement: reclaim runs only when the pool is exhausted and tests liveness
with process.kill(pid, 0), so a dead run's slot is reclaimed eventually, unless the OS has
reused the PID for any live process, in which case the slot stays parked indefinitely and
later runs are poisoned by a run that no longer exists.

Suggested fixes, independently useful:

  • On each acquisition attempt inside the same process, re-probe addresses this process
    previously retired instead of skipping them for the process lifetime. The conflict canary
    in getNextAvailableLoopbackAddress already exists to make handing out a still-held
    address safe; a freed address should return to service on the next probe.
  • Stamp pool entries with the holder's process start time (or a timestamp plus a nonce) so
    removeDeadProcessesFromPool can distinguish a dead run from a reused PID.
  • Name the still-bound ports in the teardown warning (today it prints none), so an operator
    can see in one line whether the holder is a Harper child or an unrelated service.

Reproduction

  1. macOS with no loopback aliases; HARPER_INTEGRATION_TEST_LOOPBACK_POOL_START=1,
    HARPER_INTEGRATION_TEST_LOOPBACK_POOL_COUNT=1.
  2. Ensure any listener occupies one fixed Harper port on 127.0.0.1 for the duration (e.g.
    nc -l 127.0.0.1 1883 &), so the port-release wait must expire.
  3. Run two Harper suites via harper-integration-test-run --isolation=none.
  4. Observed: suite 1 passes; teardown logs
    Harper ports on 127.0.0.1 still in use after teardown (5000ms); NOT recycling the address;
    suite 2 blocks forever in getNextAvailableLoopbackAddress with no output. Kill the run;
    the pool JSON still holds the dead PID.

Measured on

Component Version
@harperfast/integration-testing 0.7.1 (dist/harperLifecycle.js, dist/loopbackAddressPool.js)
harper (system under test) 5.2.1
Node v24.16.0
OS macOS 26.5.2 (arm64, Darwin 25.5.0), stock loopback (127.0.0.1 only)
suites datadog-agent-binary bbeb99a integration tests, --isolation=none

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions